notes on medical education assessment

KMTC Dept. of Medical Education
AssessmentAssessment
modulemodule
Norbert Boruett
A Pragmatic approachA Pragmatic approach
With Norbert Boruett

assessment units
Units in the assessment module
Unit one – principles of assessment
Unit two- concepts associated with
assessment
Unit three- assessments instruments
Unit four -Integratory

assessment units
Assessment- try to visualiseAssessment- try to visualise
why should we assess?
 what should we assess ?
 How should we assess and
 when should we assess?

assessment units
Effects of assessment on learning
and teaching
 Assessment influences what is taught and
what is learned. Often teaching is
artificially divide from assessment. The
two teaching and learning are linked by
feedback(Harden,1992).
. We shall discuss more on feedback on
subsequent parts of this module

assessment units
Effects of assessment
Torrance (1993) asserts that changing
theories and methods of assessment have
been the focus of significant attention for
some years now. Curriculum developers
have realised that real change will not take
place in schools if traditional paper and
pencil tests, be they essay or multiple
choice, remain unchanged to exert a
constraining influence on how teachers and
pupils approach curricula.

assessment units
Aschroft and Foreman- Peck (1994) point
out that assessment is a key element in the
effective management of learning than any
component of the learning process.
Curriculum developers have noted that
positive change will not take place in
learning institutions if assessment does not
change (Tornance,1993).
and teaching- cont

assessment units
Miller (1976) puts it more succinctly by asserting
that to change the curricular without changing
examination will achieve nothing.
 He further observes that changing the
examination systems without changing the
curriculum had a more profound impact upon the
nature of learning than changing curriculum
without altering the examination.
and teaching- cont

assessment units
and teaching- cont
The bicycle is a useful model when
considering assessment and teaching.
Teaching and learning are represented by
front wheel, assessment by the rear wheel.
Disasters may occur when the two wheels go
in different directions. If the rear wheel is
omitted altogether only one wheel remains:
difficulties and problems arise. In the same
way problems may arise if a curriculum is
designed with no assessment component

assessment units
Assessment and evaluation-?.
Aschrof and Foreman – Peck (2000)
highlight that assessment is often confused
with evaluation. Assessment can be
conceptualised to mean judgment about the
progress (formative assessment) or
achievement (summative assessment) of a
particular course of study or a particular
teaching intervention (Nayar, 1995).

assessment units
Formative assessment
There is a problem with formative
assessment since continuous assessment is
frequently used summatively.
Breaks learning into manageable units
Allows repeated attempts to master the
content of each module
Is not perceived as threatening (low stakes)

assessment units
Summative assessment
Is the end point examination
Can block intended career
progression(high stakes)
Is perceived as threatening

assessment units
Criterion and normative
assessment
Assessment can be seen as criterion or
normative based. Brown (1994) postulates
that criterion based assessment tells us what
has been achieved. He further articulates
that this type of assessment should have a
description telling us what has not been
achieved and the reason behind the failure.
Put, succinctly criterion referencing is
designed to check what a student has
mastered in an area of knowledge or skill.

assessment units
Tools in assessment and
evaluation
Brown (1995) asserts that the same tools
may be used to assess and evaluate at the
same time, for example it is possible to use
one tool to evaluate a teaching method and
assess a student. If a student does well in an
examination, it follows then that the
teaching method was appropriate. This
example implies the function as an
evaluation of the teaching received.

assessment units
Conclusion
As a principal objective of medical
education is to produce a competent
physician , then , unquestionably , the
raison d'être for the evaluation process is to
assess the competency and not the rank
order of students. To this end, criterion –
reference testing is necessary and must
become the principal method of evaluation
within medical education( Turnball,1989)

assessment units
Who designs an administers the
assessment ?
Traditionally the teacher has been
instrumental in executing this task.
 Perhaps you may now be aware that
currently there exist assessment methods
where students take sole responsibility and
much-pronounced role in their assessment
Examples here include portfolio-learning,
self-assessment, peer assessment (Brown,
1992; Boud, 1995).

assessment units
Assessing ability- competence
Hubbard adage states that:
There is something more scarce,
Something finer far,
Something rarer than ability.
It is the ability to recognise ability

assessment units
OSCE-as an answer
Lumley (etal., 1999) assert that OSCE
brings a new dimension to assessment of
medical training. Of particular importance
is the ability to examine practical and other
skills in a unified, measurable and
reproductive fashion.

assessment units
OSCE CONT
They are effective in assessing knowledge
and practical skills and ensure that each
student is presented with the same material,
thus providing a uniform evaluation with
the consistent marking of all those involved.
Do you recall the Kenyan experience in
OSCE?

assessment units
Why OSCE’S?
Construct validity is the ability of the
OSCE to differentiate students ability , or
follow a student’s progress before and after
a course of instruction.
Content (criterion- based) validity assesses
the value of the station in reaching its
specified objective.

assessment units
Why should students beWhy should students be
assessed?assessed?
 Safeguarding the public
 Feedback to the students
 Certification
 Monitoring the programme

assessment units
Feedback
Rowntree has laid down a claim that feedback,
or knowledge of results, is the lifeblood of
learning.
 Feedback is valuable – hence it should not
consist of notification of pass or failure.
Feedback should be given immediately and
comprehensive at the end of an examination
(Dunnington et al., 1992). Feedback should be
given positively (Amiel and Lunenfeld, 1992).

assessment units
Guidelines for giving feedback
Timing of feedback
· . Methods of providing feedback
⇒ Students checked their answers against
the correct ones written on the board.
⇒ Students discussed the questions and
answers with an instructor.
⇒ Students were assigned to textbook
materials relating to questions they had
answered incorrectly

assessment units
Feedback methods cont
 Give student’s checklist with examiners
ratings
 Watching video tape of own performance
 Discussion with examiner
 Watching video tape of expert
 Receiving feedback from patient or
standardised patient
Reading computer generated report of
performance

assessment units
What should we assess?
 Knowledge (Cognitive)
 Skills (Psychomotor)
 Attitudes (Affective

assessment units
How should we assess?
 Validity- ensure the examination is valid
 Reliability- ensure that the exam is reliable
 Feasibility- set a realistic exam that can
measure what you want to.

assessment units
When should students be
assessed?
 During the course of training
 At the end of the course

assessment units
Formative assessment
The scoring of formative assessment
should be strictly criterion referenced so as
to indicate to students their level of
performance. Aschrof and Foreman – Peck
(2000) asserts that formative assessment
should support student learning , records of
achievement , learning conflicts, portfolio
record of competence.

assessment units
quip
Examinations are formidable even to the
best prepared , for the even the greatest fool
may ask more than the wisest man can
answer.
Charles Caleb Colton ,1780-1832

assessment units
Concepts associated with assessmentConcepts associated with assessment
 Validity- measuring what you purport
(want)to measure
 Reliability- consistency.

assessment units
Types of validity
Face validity-
Content validity
Predictive validity
Consequential validity
Construct validity

assessment units
Type of validity Test facet being
measured
Questions asked
Face validity Compatibility with
educational philosophy
What is the test face
value? Does it match up
with educational
intentions?
Content validity The content of curriculum Does the test include a
representative sample of
the subject matter?
Construct validity The ability to
differentiate between
groups with known
difference in
ability(beginners versus
experts
Does the differentiate at
the level of ability
expected of candidates at
the stage in training?

assessment units
Type of validity Test facet being
measured
Questions asked
Predictive validity The ability to predict an
outcome in the future eg
professional success after
graduation
Does the test predict
future performance and
level of competence?
Consequential validity The educational
consequence of the test
Does the test produce the
desired educational
outcome?

assessment units
Reliability
Inter-rater reliability- which correlates the
consistency of rating of performance across
different examiners
Inter-case reliability which quantifies the
consistency of performance of candidate
across the cases

assessment units
Reliability
Parallel form reliability
Test retest reliability

assessment units
Estimating reliability
Split half
Cronbach alpha coefficient
 how can we handle reliability issues?

assessment units
Norm referencing and criterion
 what is the difference ?

assessment units
Why use norm referencing tests?
As you answer that - think of the inherent
dangers of norm referencing
Outline some of them

assessment units
Criterion-referenced tests
How is this different from the former? And
what are the advantages and disadvantages
of this referencing visa –vis the former

assessment units
What are the practical issues od
assessment design
Utility=
reliabilityXvalidityXfeasbilityXacceptabilit
yXeducational impact- Cees van der
Vleuten 1996

assessment units
References
1.Talpot M. Monkey see and monkey do: a
critique of the competency model in
graduate medical education. Med Edu
2004:38 587-92
2.Wass V,Vleuten CPM van der,Shatzer
J,Jones R. Assessment of clinical
comptence.Lancet 2001: 357 945-9

assessment units
Assessment instruments

assessment units
Types classroom assessment
Placement assessment
• Screening assessment
• Formative assessment
• Summative assessment
(Berk R A, 2002)
Are the students qualified, to advance to the
next, course level?

assessment units
Millers Pyramid
Types of classroom assessment
Placement assessment
• Screening assessment
• Formative assessment
• Summative assessment
(Berk R A, 2002)

assessment units
Exercise
Reflect for a moment and write down
assessment instruments that you
know and how they influence your
learning!
Remember difference between method and
instrument:
Method of travel = by land, sea, air…
Instrument of travel = train, bus, ship,
plane…

assessment units
Taxonomies
Classification of objectives or skills
• Hierarchic
– To achieve higher level one must
master lower one
• Motivation for holistic form of
education

assessment units
Bloom’s taxonomy

assessment units
Millers pyramid

assessment units
Validity
Proper assessment instrument?
– Thermometer -> Temperature
– Thermometer x-> Weight
– MCQ -> Know & understand
– MCQ x-> Show, demonstrate,
do

assessment units
• Content (=direct) validity
– Relevance?
– Representative?
– Subjective!
• Construct (=indirect) validity
– Test result -> Competence?

assessment units
Reliability
Test-retest reliability
• Equivalent forms reliability
• Split-half reliability
• Scorer/rater reliability
– Inter- vs. intrarater reliability

assessment units
Factors in clinical skills assessment that lead to
lower reliability
Two few stations or too little testing time
Check list that do not discriminate (too
easy or too hard)
Unreliable patients or inconsistent role
players
Idiosyncratic examiners
Administration problems

assessment units
Engaging with data
Write down some observationsabout the
data. What do you see?
2. Suggest explanations (hypotheses)
for the data.
3. How would you test the hypotheses?

assessment units
Feasibility
Cost in terms of resources and time of
Construction
• Marking
• Organisation
• Quality feedback

assessment units
Utility
• U = Utility
• R = Reliability
• V = Validity
• E = Educational impact
• A = Acceptability
• C = Cost

assessment units
Types of assessment
 Multiple Choice Questions (MCQs)
 Modified Essay Questions (MEQs)
 Patient Management Problems
(PMPs)
 Traditional essays questions
 Structured questions .filling the blanks
 True or false questions

assessment units
12 curriculum outcomes. SMS
What the
Clinician is able
to do-doing the right
thing
How the clinician approaches
Their practice-doing the thing
rightThe clinician as
professional
The right
person doing it

assessment units
What the doctor is able to do 1-7
Competence in clinical skills
Competence to perform procedures
Competence to investigate a patient
Competence to manage a patient
Competence in health promotion and disease
control
Competence in skills of communication
Competence to retrieve and handle information

assessment units
How the doctor approaches their
practice 8-10
With understanding of basic and clinical
sciences
With appropriate attitudes, ethical
understanding and understanding of legal
responsibilities.
With appropriate decision making skills
and clinical reasoning and judgment

assessment units
The doctor as a professional 11-12
Appreciation of the role of the doctor
within the health service
Attitude for personal development

assessment units
Multiple Choice Questions
They are designed to test knowledge
Outcomes assessed- knowledge of basic
skills and clinical sciences
8patient investigation
3 Patient management
4 health promotion
Clinical reasoning ands judgment

assessment units
Multiple Choice Questions
They are designed to test knowledge
Outcomes assessed- knowledge of basic skills
and clinical sciences
Clinical reasoning ands judgment

assessment units
What do multiple choices
questions measure
 Knowledge of specific facts
 Knowledge of terminology
 Knowledge of principles
 Knowledge of procedures
 The understanding and application of
facts and principles
 The understanding of procedures
Reasoning to some extent

assessment units
How to construct MCQs
 State the question positively
 Construct options that are grammatically
consistent with the item.
 Do not use MCQs when other instruments
are more appropriate.


assessment units
Multiple choice questions-
principles(rules)
 The question itself or the stem must be
appropriate
 Have one correct response to the
question
 Distracters or the incorrect responses
among the choices should be plausible

assessment units
Extended Matched Items
Management of diabetes in the community?
1.1. start on oral hypoglycemic
2. do diabetic annual review
3. three monthly blood tests and review
4. monthly blood tests and review
5. Refer to podiatrist
6. refer to hospital diabetes specialist nurse
7. Trial diet control for 3 months
8. 2start on twice daily insulin injection
9. Do fasting blood sugars
10. Do haemoglobin A/c blood test

assessment units
Extended matching
Select the most appropriate course of action for the
following patients
1. Woman, mid 20s ,14 weeks pregnant to have
glycosuria on routine resting in GP ante natal clinic
2. Woman,mid 50s ,BMI 30, just found to have fasting
blood sugar 15mmol/l,HBA a18.5%
3. South Asian woman, mid 30s complaining of fatique,
thirst and dry mouth

assessment units
Extended Matched Items (EMIs)
Purpose – test knowledge and reasoning skills
Outcomes assessed- all outcomes with an
emphasis on knowledge and clinical
science
5Clinical reasoning ands judgment

assessment units
MCQ
Multiple Choice Questions are used to tests
students’ factual knowledge and to some
extent the critical thinking of students. An
attempt is made to test higher levels of
Bloom’s taxonomy and not mere recall of
facts.
I
n
t
e
g
r
a
t
o
r
y

assessment units
 1. A cervical smear taken from a
thirty five year old lady was found to
have the following:
 Mitotic cells
 Multinucleated cells
 Loss of chromatin granules
 Cells that have lost adhesiveness

assessment units
Mcq
Which of the following will be the best
classification of this smear?
a) CIN I
b) CIN III
c) CIN IV
d) CIN V
e) CIN II

assessment units
Which of the following will be the best
classification of this smear?
a) CIN I
b) CIN III
c) CIN IV
d) CIN V
e) CIN II

assessment units
2. Tissue Processing in Histology is the most
critical stage in entire processes that go on
in a Histology lab. Which of the following
will determine the processing procedure?
Urgent diagnosis
 The element to be demonstrated
 The materials available
The skills involved
The size of tissue for examination

assessment units
MEQ
Modified Essay Questions will be used to
assess the students’ ability to integrate facts
and problem solving skills, which are
fundamental to good practice. Question
requiring consolidation of facts and critical
thinking will be asked.

assessment units
1. A tissue section was found to be
having a brown colour that was
intracellular upon staining with H/E
staining technique. The pigment was
found to be soluble in alcoholic picric
acid.

assessment units
Possible scenarios
i. What is the possible name of this
pigment?
ii. What features enabled you to identify the
pigment?
iii. What class would this pigment fall into?
Assuming the pigment was not soluble in
alcoholic acid picric acid, how would this
affect your identification

assessment units
Formatively as learning tool to stimulate
discussion and top plan future development
Can be used to stimulate and provide
feedback – may be every three months what
is going , intriguing / exciting ,worrying
phenomenon.

assessment units
What can MEQ measure ?
MEQ can be used as a formal assessment
technique
 It can be used as an aid to learning.
 It can be used as an adjunct to a lecture.
 It presents good questions for group
discussion.
 It can be used as a self-instruction method.
It can be used as a self-audit device.

assessment units
Construction of MEQ
 Drawing up a draft of the subject area.
 Circulating the draft to a team of fellow
professionals to study it.
 Each of the professional colleagues should be
asked to provide the answer to each of the
questions.
 These responses should be studied and forms
the basis for model answers.
The marks should be allocated according to
consensus decisions

assessment units
PATIENT MANAGEMENT PROBLEM
(PMP)
Patient management problem (PMP) consists of
short questions that require also short answers that
revolve around managing a patient. They are
intended to equip the student with knowledge,
skills and attitude that are necessary in
professional practice.
The main feature is that a series of questions are
asked about a case and options are provided that
leads to proper management of the patient. It tests
students on a wide range of subjects.

assessment units
What PMP can assess
Like the MEQ, PMP provides the opportunity for
students to demonstrate their ability in patient
management skills and problem solving skills in a
variety of scenario.
The students are presented with simulated
patients, real patient or imaginary patient in a
paper and pencil format. The unfolding scenario
is presented logically and the student is evaluated
on the basis of the proper steps taken leading to
the proper management of the patient.

assessment units
Constructing a PMP
Constructing a PMP begins with a
description of the symptoms of the patient
and the resources that are available. The
student is presented with the problem
followed by a series of questions relating to
management, diagnosis and investigations.
Each question has a number of options from
which the student must select. Beside each
option are hidden instructions about how
the student should proceed.

assessment units
Steps in constructing PMP
 Clearly define what it is you wish to test in the
problem. Set realistic objectives for the group you
are testing.
 Select a problem that is meaningful and
suitable.
 Ensure that the sequence of management is
logical.
 Prepare the opening scene. This should
compose of: the actual setting for the case, the
presenting features of the patient, and the role of
the user, the resources available and clear
instructions.

assessment units
Steps continue
 Prepare a list of clinical choices which
are consistent with the problem.
 Pretest the PMP using a group of subject
experts. Also try it out with one or two
potential candidates.
 Score the PMP. Allocate a mark for each
individual choice on the PMP.

Work-based Assessment
Assessing learning acquired in the
workplace

assessment units
Work-based Learning
Learning acquired from and through work
experience
Usually involves a learning contract that
outlines what the person will undertake to
achieve from the work experience
Also involves a mentor from the health
service in the workplace
Involves academic staff in assessing and
accrediting work-based learning

assessment units
Work-based Learning
Brings together educational providers and
service providers and increases relevance
Particularly useful for continuing
professional development (CPD)
Economically attractive since the person
remains in the workforce providing service
Requires academic staff to develop new
competencies in relation to supervision,
assessment, and distance learning

assessment units
The use of Work-based Learning in higher
education can facilitate the development of
situational, practical, process and people-
oriented knowledge (Eraut, 1994)
It can encourage on-going reflection on
knowledge, values and practice, thus
increasing the professional’s self-awareness
and understanding of the social factors that
affect performance, such as health service
policy, management and organisation.

assessment units
WBL requirements
The award to which the learning
contributes must be within a credit
accumulation and transfer system (CATS)
Credit must be based on achievement of
specific learning outcomes - preferably
competencies
Assessment procedures and instruments
have to be performance-based

assessment units
Assessing competencies
Assessment that draws on evidence of
professional competence:
tests a wider range of knowledge, skills
and attitudes and, also, how well they are
integrated in practice
encompasses a balance of propositional
and process knowledge through
performance

assessment units
feeds back to teaching and learning and
helps workplace mentoring by assessing
against competence criteria
allows flexibility and participation in
assessment, with students judging the right
time to be assessed
prioritises assessment that is conducted in
real work settings, providing authenticity
and feedback to the learner

assessment units
helps to make transparent the state of the art in
professional practice (through analysis of
competencies) and assists in its on-going
improvement
facilitates the setting of reliable and valid national
standards, and the marketing of marketable and
transferable awards
links in with other approaches that stimulate
reflection on education and practice (e.g. formative
assessment, profiling and records of achievement, self-assessment)

assessment units
The framework for approaches to
WBL assessment
General competence criteria set in an integrated,
holistic framework provide the basis for assessment in
professional contexts
Competence is a construct inferred from the available
evidence of skills, knowledge, attitudes and values
Authentic assessment tasks present open-ended
situations that test professional judgement

assessment units
WBL assessment
Routine performance and reproductive thinking
give way to higher-order, creative thinking that:
is complex
yields multiple solutions
requires application of multiple criteria
often involves uncertainty
fosters self-regulation in the thinking process
seeks to create meaning in experience

assessment units
WBL assessment
Need for a multi-dimensional model that
reflects the interplay in medical practice of
professional attributes that are knowledge-
based and experienced-based
 scientific knowledge, judgement, patient management skills,
technical skills, interpersonal skills, attitudes and personal
characteristics, experience monitoring and effectiveness of
health interventions
WBL assessment has to measure the
practitioner’s success in synthesising from this
repertoire

assessment units
Approaches to
WBL assessment
Will the assessment be focused on Process
or Outcome?
Will the assessment provide
Hard (quantitative) or Soft (qualitative)
measures?

assessment units
Hard - Process assessment
Frequency counts of conversations, body
language, requests for learning support
Time on task
Audit
Monitoring of computer data

assessment units
Hard - Outcomes assessment
Multiple assessors - managers, colleagues,
patients
Performance checklists/ Rating schedules
Observation/video of skills demonstration
or simulated tasks
Multiple choice tests/ Modified essays
Written report (structured)
Oral presentation/poster

assessment units
Soft - Outcomes assessment
Observation of learner attributes, motivation,
inquiry orientation
Generic skills/ transfer
Profiles
Personal/ joint reports
Self- or peer assessment

assessment units
Soft - Process assessment
Summative statement
Observation of body language, learner
interactions, consultations with
tutor/supervisor
Dialogue around learning strategies, log
books/diaries
Cognitive mapping, reflexive self-learning

assessment units
Portfolios
Best way of achieving a holistic picture
Multi-dimensional and integrate evidence
gleaned from a range of methods
Bring together the results of performance
ratings, supervisors’ judgments, and assessed
work such as project reports
Present evidence of personal reflection and
critical evaluation

assessment units
Assessing Group Work
This method compliments efforts of small group
teaching. As task is given to students and marks
awarded. They further clarify that the task may be
divided to give individual marks. There are a
number of disadvantages in this type of assess,
including:
 - Dysfunctional group dynamics
 - Cultural factors
 - Members who don’t value small group
teaching.

assessment units
Diaries
Students’ records can be kept as a
supplementary or additional form of
assessment. Ashcroft and Foreman-Peck
(2000) assert that it can be set as an
alternative to more traditional forms of
assessed writing.

assessment units
Self-Assessment
Self-assessment is being used more an
more in higher education as a strategy for
both student learning and assessment
(Boud, 1977). It is a key skill to be
developed on any course. Self-assessment is
not a new techniques, but a way of
increasing the role of students as active
participants in their own learning (Boud,
1995).

assessment units
Self assessment continued
 All assessment – whether conducted by teachers
or by learners – involves two key elements.
· The development of knowledge and appreciation of the
appropriate standards and criteria for meeting those
standards
that may be applied to any given work. Unless it is known
what counts as good work, it is impossible to tell
whether the specific work being considered is
adequate.
· The second is the capacity to make judgements about
whether or not the work involved does or does not meet
those standards.

assessment units
Peer Assessment
This is the assessment involving students.
Students give themselves qualitative
feedback to fellow colleagues during a
presentation. Feedback may also be given
on written work. Giving feedback is a tricky
affair, students may become defensive.
Students may possibly be asked not to reject
comments. Assessment should be made on
agreed criteria made by the students.

assessment units
Learning Contracts
Learning contracts may be seen as
agreement between a student and a tutor
about what and how a student should be
learning. Ashcroft and Foreman (2000)
suggest a well designed pro-forma, detailing
out what objectives are to be agreed on. The
contracts can be re-negotiated.

assessment units
Records of Achievement
Records of achievement sometimes call
profiles, are systematic attempts to involve
students in recording, reviewing and
evaluating their academic and personal
development through out the college career.

assessment units
Portfolio assessment
A portfolio is literally ‘a collection of
papers ’Portfolio based learning is a method
of encouraging reflective learning for
professions. A professional development
portfolio is a collection of materials, made
by a professional, that records, and reflects
on, events and process in that professional’s
career (Hall, 1992).

assessment units
What do portfolios offer?
They recognize and encourage the
autonomous and reflective learning that is
an integral part of professional education
and development
They are based in the real expereince of the
learner, and so help the connection between
theory and practice.
They cater for a range of learning

assessment units
What do portfolios offer?cont
They enable assessment within the
framework of clear criteria and learning
objectives.
They can accommodate evidence from
different sources.
They provide a process for both formative
and summative assessment , based on either
personally derived or externally set learning
objectives

assessment units
They provide a model for life long learning
and continuing professional development

assessment units
What is the educational rationale
 promote adult and learner cantered
learning -Portfolios provide educational
opportunities that are interactive and
reflective and can be related to real life
experience.
In many ways ,portfolios may be
considered the ultimate educational tool in
terms of meeting the criteria for good
practice in adult learning.

assessment units
Portfolio learning characteristics
Learning impacts on expertise in several
ways
Making changes in practice
Seeking answers to new questions
Validating established practices

assessment units
What does a portfolio look like?
There is no common for a portfolios range
from simple notebooks and diaries to larger
A4 folders and computer held files. Some
learners prefer to given something , and A4
szed file with a dairy inserting has been
welcomed by many . However being
prescriptive can generate resistance, and
flexibility over format is important .

assessment units
What can be included in
portfolio ?
Critical incidents or events with patients
A reflective journal or dairy
Tutorials and learning plans and reflection
on them
Exam preparation material
Video recording of consultations and other
relevant material

assessment units
What can be included in
portfolio ?
Audits and project work
Critical reviews of articles
Feedback material
Management material

assessment units
Critical incidents of events with
patients
Critical incidences are those incidents in a
working day that are memorable for going
well or badly, or which have proved
generally thought provoking(snadden et
al.1999).

assessment units
Reflective journal or diary
Putting thought into paper this encourages
the learner to reflect into what they are
doing.

assessment units
Tutorials and learning plans
Descriptions of tutorial that have happened
and the follow up of learning needs that
have resulted from them can be used.
The areas of strengths and developments
are noted and future references can be made
to see if learning has taken place

assessment units
How can the material in a
portfolio be used?
As a method of promoting personal
development
Formatively as a learning tool to stimulate
discussion and to plan future learning
As a formal (Summative) assessment tool.

assessment units
As a method of personal
development and way of tracking
progressPortfolios can be used for self learning
activity without support and input from a
tutor or mentor.
Learners collect material to track down
their own progress
Network of tutors and mentors would
make the learning more sucessful

assessment units
Assessing portfolios
Assessing portfolios presents a number of
particular challenges because of the highly
individual nature of the portfolio.
The process does not fit in the tradition concept of
assessment in medical education.which has its
roots in the science of objectivity.
In portfolio we must have room for subjectivity
and artistic appreciation of the work presented

assessment units
Portfolios assessment may have the following
aims:
To provide feedback to learners so that can
learn from mistakes and build on
achievements
To motivate learners and focus their sense
of achievements
To enable learners to correct errors and
remedy deficiencies
To consolidate learning

assessment units
aims:
To help learners to apply abstracts
principles to practical contexts.
To guide selection,option or career choice
To classify or grade learner achievements
To estimate potential to progress to other
levels or courses
To give teachers feedback on how effective
they are at promoting learning

assessment units
aims:
To provide statistics for internal and
external agencies

assessment units
aims:
To give teachers feedback oh how effective
they are promoting learning
external agencies(Brown and Knight,
1994).

assessment units
Strengths of portfolios
Portfolios have their strengths in providing
a method of giving feedback, in helping
learners define their strengths and weakness
and in providing evidence of achievement
towards set learning objectives. These
objectives can be set by the learner or by an
external body(Snadden et al.,1999).

assessment units
As a formal (summative ) assessment tool
Assess practice over a period of time- it
assess performance in practice over a period
of time. Authentic assessment.
Useful in formal and summative
assessment in nursing(Jasper,1995).
Snadden and Thomas ,1996: 1998point out
the negative effect of using portfolio as an
assessment tool.

assessment units

assessment units
aims:
To give teachers feedback oh how effective
they are promoting learning
external agencies.

O S C EO S C E

assessment units
What is an OSCEOSCE ??
OObjective SStructured CClinical EExamination
ObjectiveObjective – means fair and without
bias. Most examination in the world
are not fair. Use of checklist
ensures objectivity.What is your
comment?
StructureStructuredd refer to the organization
of the examination. Why use OSCE
. The rationale.

assessment units
O.S.C.E. cont.
Globally many patients are gaining rights
and becoming fewer. Thus students may
not gain meaningful clinical experience.
During examination periods students get to
be examined in different and sometimes
cases that do not pose the same difficulty.
What would be the effect?

assessment units
 This type of examination is characterized by severalseveral
stationsstations, that reflect aspects or areas a clinician as to
master. Think of some roles a doctor/ nurse must be able
to do.
 ClinicalClinical – the all examination entails the clinical aspects
of a an health worker. In other words the station are
clinical in nature.
 ExaminationExamination – this summarizes the all episode. It is an
examination with usually declares those who are safe to
handle the public. Larger market share
O.S.C.E. cont.

assessment units
Goal and Objective of anGoal and Objective of an
OSCEOSCE
 Produce competent health workers
 Help in the learning process of the
students and a humanistic assessment
method
 Allow for meaning research and staff
development.

assessment units
WHY OSCE
Validity issues – most examinations do not measure
What they purport to measure. Think of many theory
examination that we really on
Reliability – consistency many examination set ups
are not consistence in the way it determines failures
and those who have performed well

assessment units
Typical observer sheet- rating scale/ check list
must have the:
Name of std,skill, tasks
Competence Yes No
Greets patient
Explains procedure
Assembles equipment
Withdraws medicine and expels air
Swaps the area the correct area
Inserts needle horizontally and gently
propels medicine

Observer sheets- Weighting the
skills
A mark is always provide against any
correct maneuver. However some
skills must be weighted heavily
because of their importance

assessment units
Scoring the skill
Add all the correct procedures
Double the score as and when indicated
Divide the total score obtained against the
total possible score
Score= correct score
Total possible score

assessment units
OSCE
Requires a lot of
tedious planning
Requires skilled
personnel
Requires substantial
resources
Demands a lot of
dedication on the part
of the participants

Integratory
unit

assessment units
Integratory
This integral unit covers various
assessment instruments and an elaborate
procedure of developing Objective
Structured Clinical Examination
(OSCE) is also discussed.
I
n
t
e
g
r
a
t
o
r
y

assessment units
Assessment and learning outcomes
Many medical schools are developing
learning outcomes or goals that expect their
graduates to achieve at the course often
termed exist outcomes.
They be expressed as competencies, roles,
outcomes or goals or knowledge, skills and
attitudes

assessment units
Assessment and learning outcomes-
cont
The wording of learning objective give glues to
the type of assessment instrument to be used.
Recall ,name, define, state, describe, be aware, understand’
refer to knowledge and therefore are generally assessed by
knowledge based examination such as MCQ, extended
matched items.
 explain recognize ,review, discuss, identify, distinguish,
debate, analyse, assess design, refer to the interpretation of
knowledge and ability to reason- this are assessed by EMI,s,
structured responses questions essay and project work

assessment units
Assessment and learning outcomes-
cont
The wording of learning objective give
glues to the type of assessment instrument
to be used.
Demonstrate ,use ,apply, perform, show, be
able to, refer to skills that you need to
acquire that may be assessed in OSCEs, long
cases or OSLERs.

assessment units
TheThe OSCE ProgrammeOSCE Programme

assessment units
TheThe OSCEOSCE ProgrammeProgramme
What is an OSCE ?
Norbert BoruettNorbert Boruett

assessment units
OSCE
An examination
gaining global appeal

assessment units
What is an OSCE ?
Objective Structured Clinical Examination
Objective – means fair and without
bias. Most examination in the world
are not fair. Use of checklist
ensures objectivity.What is your
comment?
Structured refer to the organization
of the examination. Why use
OSCE?. The rationale.

assessment units
 Clinical – the all examination entails
the clinical aspects of an health
worker. In other words the station are
clinical in nature.
 Examination – this summarizes the
all episode. It is an examination with
usually declares those who are
competent to handle patients.
O.S.C.E. cont.

assessment units
Goal and Objective of an OSCE
 Help in the learning
process of students
 Produce competent
health workers
 Allow for meaning
research and staff
development.

assessment units
Think?
WHY OSCE ?
 Validity issues – most examinations do not
measure WHAT they purport to measure.
 Reliability – consistency many examination
set ups are not consistence in the way it
determines failures and those who have
performed well

assessment units
Station
 This is the region where the
skill is demonstrated by the
candidate
 The assessors check ()
against the competence
shown in check list.
 i.e. giving an injection

assessment units
Station RotationStation Rotation
Station 1Station 1 Station 2Station 2
Station 3Station 3
Station 4Station 4
Station 5Station 5Station 6Station 6
Station 7Station 7
ININ
OUTOUT

assessment units
The Observer Sheet
Norbert BoruettNorbert Boruett

assessment units
Typical observer sheet: Name,skill, tasks
Competence Yes No
1. Greets patient
2. Explains procedure
3. Assembles equipment
4. Withdraws Medicine and expels air
5. Swaps the correct area
6. Inserts needle horizontally and gently
7. Propels medicine








assessment units
Observer Sheets
Weighting the skills
 A mark is always provided against any
correct maneuver. However some skills
must be weighted heavily because of their
importance!

assessment units
Scoring the skill
 Add all the correct procedures
 Double or triple the score as and when
indicated
 Divide the total score obtained against the
total possible score
 Score=correct score / Total possible
score

assessment units
How to Design an OSCE?

assessment units
Design and Run an OSCE?!
Teamwork!

assessment units
Teamwork!
Brainwork!

assessment units
Teamwork!
Brainwork!
Think work!

assessment units
Teamwork!
Brainwork!
Think work!
Venture!

assessment units
Teamwork!
Brainwork!
Think work!
Venture!
Tour-de-Force!

assessment units
BUT…
It is a real Challenge!
It is absolutely Worthwhile

assessment units
BASIC CRITERIA
For the Ideal Skills Assessment
 To be Well Founded and Sound-Based
 To have Validity and Reliability
 To have a Practical Manageability
The Assessment has:The Assessment has:

assessment units
BASIC CRITERIUM 3
Practical Manageability
 The number of Students to be assessed?
 The available Staff Members
 The available Workspace,
accommodation & requisites
 To what extend the obtained assessment
results can be used in an objective way
 To let pass or let fail the student
 To filter out the poor or the bad student

assessment units
Step 1Step 1
The Scenario for the OrganisationThe Scenario for the Organisation
Of Typical OSCE SessionOf Typical OSCE Session
Step 2Step 2
Step 3Step 3
MonthsMonths
WeekWeek
DayDay
Preparatory TasksPreparatory Tasks
The Week- & Day-The Week- & Day-
Before TasksBefore Tasks
The Day-Itself TasksThe Day-Itself Tasks

assessment units
Step 1Step 1 Preparatory TasksPreparatory Tasks
 Fix dates of the OSCEFix dates of the OSCE  day(s)day(s)
 StaffingStaffing  ObserversObservers
 Test SubjectsTest Subjects
 Students to pass the OSCEStudents to pass the OSCE
 Accommodation facilitiesAccommodation facilities  RoomsRooms
 ManikinsManikins
 AttributesAttributes

assessment units
Step 1Step 1 Preparatory TasksPreparatory Tasks
 Identify the StationsIdentify the Stations  Number of stationsNumber of stations
 Skill in each stationSkill in each station
 Design Observer SheetDesign Observer Sheet
 Consent on gradingConsent on grading
 Secretarial TasksSecretarial Tasks  AnnouncementsAnnouncements
 SubscriptionSubscription
 Control systemsControl systems
 Roster designsRoster designs
 Task distributionTask distribution

assessment units
StepStep 22 Week & Day Before TasksWeek & Day Before Tasks
 Are All Students InvolvedAre All Students Involved properly Informed?properly Informed?
 Suspension of all Classes, lecturers, practicals etc?Suspension of all Classes, lecturers, practicals etc?
 Is Administration and Direction fully informed ?Is Administration and Direction fully informed ?
 Are All OSCE Staff Member committed to theirAre All OSCE Staff Member committed to their
several tasks?several tasks?

assessment units
StepStep 22 Week & Day Before TasksWeek & Day Before Tasks
 Organise the SET of the OSCEOrganise the SET of the OSCE
 Station Rooms, Requisites, Manikins,Station Rooms, Requisites, Manikins,
Waiting RoomsWaiting Rooms
 Security of the Content of the StationsSecurity of the Content of the Stations
 Final Rehearsal en BriefingFinal Rehearsal en Briefing
 Programme in minute detail is distributedProgramme in minute detail is distributed

assessment units
StepStep 33 The Day Itself TasksThe Day Itself Tasks
 GENERAL RULEGENERAL RULE
 An OSCE Runs on the StopwatchAn OSCE Runs on the Stopwatch
 All students present 15 minutes beforeAll students present 15 minutes before
launchlaunch

notes on medical education assessment

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie notes on medical education assessment

Ähnlich wie notes on medical education assessment (20)

Mehr von Amref International University

Mehr von Amref International University (6)

notes on medical education assessment

Hinweis der Redaktion