1 blueprinting

SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting and drafting
questions
Liz Norman
Massey University
Sharanne Raidal,
BVSc, PhD, FANZCVS, GradDipEd
http://consultatutorblog.files.wordpress.com/2010/06/examstress.gif
Australian and New Zealand College of Veterinary Scientists
Examiner Workshop, 6 – 7th
February 2016

Overview
• Blueprinting and drafting questions
• Grading criteria and marking schemes
• Clarity, timing; oral examinations
http://www.slidehare.net/liznorman
http://toonclips.com/600/4997.jpg

What is a blueprint?
blueprint, n. something which acts as a
plan
model or
template
OED Online http://www.oed.com https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg

What is a blueprint?
• Specifies what’ going to be in the exam
 Content / topics (breadth)
 Level (depth)
https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg

Purpose of blueprinting
• To document
sampling
• To show that your
examination is
representative of all
that could be
examined and is
sufficient
Content domain
Exam
2
Oral
Essay
Credentials
domain
Exam
1
Prac
Practice domain

Why blueprint?
• To show that we
have a
representative
sample
• To permit
extrapolation to
content (practice)
domain
• To plan how each
component assessed
Content domain
Exam
2
Oral
Essay
Credentials
domain
Exam
1
Prac
Practice domain

Content of questions
• Scope of topics
• Other relevant considerations - eg. species
body systems
• Scope of skills and abilities

Subject guidelines
• Represent the entire content domain
(ie. they define all that is examinable)
• Specify both scope (breadth) and level (depth) of
the knowledge, skills, attitudes and judgements
required
• For Fellowship, some aspects are assessed in
the credentials document (and process and
training)

Subject guidelines
The candidate will have a detailed knowledge of:
The aetiology, pathogenesis and
pathophysiology of cardiac, renal, respiratory,
alimentary, musculoskeletal, endocrine,
ophthalmological and neurological organ
dysfunction in the cat and the dog.

Level of questions; skills & abilities required
Level – depth
 Levels of knowing and understanding
Thinking processes required by the discipline
•Surface vs deep
•Fact recall vs application
•Blooms taxonomy
•SOLO taxonomy

Fact recall:
Questions capable of being answered by reference to one
paragraph in a text or notes (or several paragraphs for
questions requiring several facts).
Applied (higher order):
Questions that require the use of facts or concepts, the
solution of a diagnostic of physiologic problem, the
perception of a relationship or other process(es) beyond
recalling discrete fact.
From: Peitzman et al. (1990) Academic Medicine , 65(9): S59-60

Bloom’s taxonomy
http://julietovar.edublogs.org/files/2011/05/blooms-taxonomy-1k4snjn.JPG

Bloom’s taxonomy
http://www.psia-nw.org/wp-content/uploads/Blooms_Taxonomy.jpg
Recall
Application

Bloom’s instructional verbs
Create: compose, plan, propose, design, formulate, arrange, assemble, collect,
construct, create, set-up, organise, manage, prepare.
Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate.
Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test,
compare, contrast, criticise, diagram, inspect, debate, question, relate, solve,
examine, categorise.
Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate,
operate, schedule, sketch.
Understand: translate, restate, discuss, describe, recognise, explain, express,
identify, locate, report, review, tell.
Remember: define, repeat, record, list, state, recall, name.
Recall
Apply

http://www.pescholar.com/wp-content/uploads/2013/03/Solo-SS.png

SOLO taxonomy
http://litre.ncsu.edu/solowrittencomposite.gif

Level of questions skills & abilities required
SOLO taxonomy
http://www.mbms.org.uk/faculties/science/assets/SOLO/solo-
taxonomy-explained-full.jpg

SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely
misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response
(such as being able to describe aetiology, clinical features,
management) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect
relationships; able to compare similarities and differences
between apparently distinct phenomena; this level is taken as
suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena
and apply this information to a novel context – able to develop
novel hypotheses, theories and deduce principles; creative
thinking

SOLO taxonomy
thinking
Quantitative change
Qualitative change

SOLO taxonomy
thinking
Deep
Surface

SOLO taxonomy
thinking
Application
Recall

SOLO taxonomy
Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple
instructions, calculate, reproduce, arrange, recognise
Multistructural
Combine, classify, describe, enumerate, list, do algorithm,
apply method, account for, execute, formulate, solve, conduct,
prove, complete, characterise
Relational
Analyse, compare, contrast, integrate, relate, explain causes,
apply theory, argue, implement, plan, summarise, construct,
design, interpret (some senses), structure, conclude,
substantiate, exemplify, derive, adapt
Extended abstract
Theorise, generalise, hypothesise, predict, judge, transfer
theory (to new domain), assess, evaluate, interpret (some
senses), critically reflect, predict, criticise, critique, reason

Skills and abilities
•What is contained in subject guidelines?
•Skills beyond cognitive?
•Recognition / interpretation?
•Communication?
- diagnostic images
- cytology / pathology
- reports
- statistics

Blueprinting - process
• Mapping to scope and level

• Breadth by learning outcome
Written paper
1
Written paper
2
Practical
examination
Oral
examination
LO1 x x x
LO2 x x
LO3 x x x
LO4 x

• Breadth by topic
Patho-
physiology
Investigation &
diagnosis
Treatment &
management
Gastrointestinal P1Q1 P1Q1, P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5

• Breadth by topic and depth
Patho-
physiology
Investigation &
diagnosis
Treatment &
management
recall higher recall higher recall higher
Gastrointestinal P1Q1 P1Q1 P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous
P1Q3,
P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5

• Breadth by species
Species Number of Qs Percent
Small animal 15 52%
Farm animal 4 14%
Horse 5 17%
Exotic 3 10%
Lab 1 3%
All 1 3%

• Content coverage x depth
Topic 1
30%
Topic 2
30%
Topic 3
40%
Total
Knowledge of terms 2 5 5 12
Comprehension of principles 4 3 4 11
Application of principles 3 3 3 9
Analysis of situations 3 2 5 10
Evaluation of situations 3 2 3 8
Total questions 15 15 20 50
http://scoring.msu.edu/written.html

Blueprinting
Enables you to plan and check that your exam, or
set of exams, covers the content (and skills) it is
meant to cover, at an appropriate level.
Without this you cannot extrapolate performance
on this exam to performance in the whole domain.
Validity

Controlling difficulty and demand
Should questions be difficult?
Norm-referenced vs
standards-based grading

Should questions be difficult?
Appropriate demand
vs irrelevant difficulty
Practice domain
Content
domain

What makes questions difficult?
•Familiarity / novelty
•Complexity – components, links
•Scale – components, links
•Resources utilised (provided, generated)
•Abstraction – concrete experience, concepts, principles
•Task strategy – simple, stepwise, integrated
•Guidance
Adapted from Hughes et al (1998) The development
of a tool for gauging demands of GCSE and A level
exam questions. Brit Ed Res Assoc Conference.

Familiarity / novelty
Question 1
Explain the physiological actions of insulin.
Question 2
Explain the physiological actions of ghrelin.

Familiarity / novelty
Question 1
List four (4) clinical signs of diabetes mellitus in dogs.
Question 2
List four (4) clinical signs of feline hypersomatotropism.

Complexity, scale – components, links
Question 1
Compare and contrast the clinical signs of
hypoadrenocorticism with those of hyperadrenocorticism in
dogs.
Question 2
Compare and contrast the clinical signs of diabetes mellitus
with those of hyperadrenocorticism in dogs.

Complexity, scale – components, links
Question 1
Compare and contrast the clinical signs of
diabetes mellitus with those of hyperadrenocorticism in dogs
and cats.
Question 2
Compare and contrast the clinical signs of diabetes mellitus
with those of hypoadrenocorticism in dogs and cats.

Resources utilised – provided, generated
Describe the skull.
(30 marks)

Abstraction – concrete experience, concepts,
principles
Discuss the concept of quality of life in terms of biological
function, “feelings” and natural existence.
Describe how both classical conditioning and operant
conditioning are involved in cows confidently entering the
milking shed and letting down.

Task and response strategies – simple, stepwise,
integrated
Question 1
Discuss the use of insulin for the treatment of diabetes
mellitus in cats.
(25 marks)

integrated
Question 2
a)Describe the advantages and disadvantages of insulin therapy for
diabetes mellitus in cats. (10 marks)
b)Indicate the dose and frequency of administration of insulin you
would prescribe to a newly diagnosed cat with diabetes mellitus. (5
marks)
c)Describe the recommendations you would make for the frequency
and timing of feeding in relation to insulin dosing in cats with
diabetes mellitus. (10 marks)

integrated
Guidance
Describe the skull. (30 marks)

integrated
Guidance
You are presented with a 7 year old male Warmblood gelding with exercise
intolerance. You auscultate an arrhythmia with a variable pulse and order an
ECG. Define the type and give a specific name for the arrhythmia demonstrated
in the rhythm strip below. (2 marks)

integrated
Guidance
You have been contracted by a farmer producing Pacific oysters (Crassostrea
gigas) intertidally, in a bay containing a number of oyster farms. The farmer is
concerned with the amount of dead shell they are seeing during the current
grading. Explain how you would approach this scenario. (20 marks)
Include in your answer how the information you could gather might influence
your assessment, what differential diagnoses you consider, and detail how you
might further investigate potential causes, and what advice you would provide.

What’s different about oral questions?
• Thinking on your feet
• Can’t revisit or edit
• Tests verbal communication skills (as well as
knowledge)
• Interactive – can prompt and redirect, can ask
candidate to explain their answer

Key points

Blueprinting
• Ensure the examination (as a whole)
representatively samples from the domain
specified for the examination
• Membership – this is all the learning outcomes
• Fellowship – this is all the learning outcomes,
except those that can only be sampled in the
credentials document

Blueprinting
• Ensure the examination (as a whole)
representatively samples from the domain
specified for the examination
• Consider - topics
- level
- species, body systems, modalities
- skills
- other things
(eg. Dx vs mgt, principles vs application)

Structuring the exam
• Ensure you follow subject guidelines explicitly
• Shorter vs longer questions
• Scenario vs direct questions
• Difficulty, guidance

Next session’s task:
• Check your blueprint - content
- level
• Classify your draft Qs according to SOLO
taxonomy
• Redraft Qs if necessary

1 blueprinting

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (12)

Ähnlich wie 1 blueprinting

Ähnlich wie 1 blueprinting (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

1 blueprinting