This document summarizes a study on design judgment in instructional design practice. The researchers observed 8 instructional designers and identified different types of judgments they made. They found that judgments were made continuously throughout projects and were highly situational based on contextual factors. Design judgments occurred in complex, layered ways and involved multiple types of judgments. The researchers believe this study suggests instructional design education should focus less on teaching isolated tools/models and more on helping students develop their ability to make judgments within complex design contexts.

1. Stop Telling Designers
What To Do:
Reframing Instructional
Design Education Through
the Lens of ID Practice
elizabeth
boling,
colin
m.
gray,
and
muruvvet
demiral
uzan

2. additional
collaborators
- Cesur
Dagli
- Funda
Ergulec
- Abdullah
Altuwaijri
- Khendum
Gyabak
- Megan
Hilligoss
- Remzi
Kizilboga
- Verily
Tan
- Kei
Tomita

3. background,
purpose,
&
previous
studies
- Research
on
ID
practice
has
been
limited
- On
its
own
terms
(Rowland,
1992)
- Comparing
practice
to
existing
ID
models
or
frameworks
(e.g.,
ADDIE)
(Wedman
&
Tessmer,
1993;
Visscher-­‐Voerman
and
Gustafson,
2004)
- Attempts
to
approach
the
complexity
of
practice
by
translating
it
into
explicit,
teachable
terms
(Ertmer,
York,
&
Gedik,
2009;
Ertmer
et
al.,
2009)
- Lack
of
knowledge
about
practice
has
constrained
our
conceptions
of
what
ID
education
should
include
- transferring
an
idealized
notion
of
design
to
students,
providing
learners
with
knowledge
of
the
models,
theories,
and
principles
advocated
by
the
field
(Smith,
2008;
Smith
&
Boling,
2009)

4. design
practice
&
judgment
- Conception
of
judgment
in
the
broader
study
of
design
- based
in,
or
impelled
by,
tacit
knowledge
that
affects
the
actions
of
designers
(see
Polanyi,
1966;
Vickers,
1984;
Holt,
1997)
- Related
to
modes
of
cognition
and
forms
of
knowledge
specific
to
designing
(Cross,
2010)
- comprises
multiple
forms
that
are
(Nelson
&
Stolterman,
2012):
- exercised
as
nuanced
decisions
throughout
designing
- not
limited
to
rational
choice
of
strategies
based
in
data

5. design
judgment
types
Type Operationalized
Definition
framing Creating
a
working
area
for
design
activities
to
occur,
often
by
introducing
constraints
(client
or
tool)
or
ways
of
assessing
outcomes.
This
occurs
dynamically
across
multiple
levels.
deliberated
off-­‐hand Recalling
to
consciousness
previous
judgments
that
have
led
to
successful
practices
and
opening
them
to
the
possibility
of
adaptation
or
use.
appreciative Placing
high
value
and
emphases
on
certain
aspect/s
of
a
design
situation
while
backgrounding
others.
quality Making
design
decisions
about
the
effectiveness
of
visual
and
other
forms
of
style,
or
to
demonstrate
due
diligence,
often
in
accordance
with
company
standards,
in
relation
to
a
concrete
design
artifact.
appearance Assessment
of
overall
quality,
relating
to
an
entire
product
or
experience,
rather
than
just
a
portion.
This
often
includes
part/whole
relations
within
a
frame
of
aesthetic
experience
or
measurement
against
heuristic(s).
connective Making
connections,
or
bridging
various
design
objects
that
are
central
to
the
design
process
and
activity.
The
connections
made
in
this
context
are
not
generalized
but
specific
to
the
design
situation.

6. design
judgment
types
Type Operationalized
Definition
compositional Making
connections
or
bringing
various
design
objects
together
that
are
central
to
the
design
process
and
activity.
The
connections
made
in
this
context
are
generalized
and
not
specific
to
a
particular
design
situation
but
to
the
overall
process.
instrumental The
selecting,
utilization,
or
influence
of
a
tool,
concept,
or
method
in
reaching
an
established
design
goal.
navigational Considering
a
path,
plan,
or
certain
manner
(of
individual,
disciplined
preference)
in
approaching
a
task
or
a
challenge
to
get
to
a
desired
state.
default Giving
an
automatic
response
to
a
situation
without
deliberation.
core Statement
about
one’s
value
or
thinking,
usually
revealed
when
pushed
by
“why”
questions
concerning
one’s
judgment.

7. research
questions
&
method
- What
do
IDs
do
in
practice
consistent
with
design
judgment?
- What
design
judgments
take
place
in
ID
activities?
- exploratory
questions
and
research
design
- 8
practicing
IDs
at
two
sites
of
ID
practice
- field
observations
(20
hours
total)
with
handwritten
field
notes
and
follow-­‐up
interviews
with
notes
and
audio
recording
-
unitized
coding
of
judgments
- holistic
case
summaries
- we
observed
IDs
in
whatever
part
of
their
projects
were
happening
at
the
time
we
were
there
…
we
did
not
assume
judgments
would
be
happening
only
at
certain
times,
and
we
were
not
assessing
or
describing
any
specific
project

8. Name
Company
Role
Years
of
Experience
Background
Gabriel
Campus-­‐wide
Consultancy
Media
Consultant
6
Degree
in
Computer
Science;
Masters
in
Comm.
&
Tech.
Emily
Established
ID
Firm
ID
0.17
ID
in
non-­‐profit
fields
Julia
Established
ID
Firm
Senior
ID/Project
Leader
7
Degree
in
Fine
Arts;
Masters
in
IST
Heather
Established
ID
Firm
Project
Manager
11
Degree
in
English;
Masters
in
IST
Ethan
Established
ID
Firm
ID
2.5
Degree
in
IT;
Masters
in
Instructional
Tech.
Claire
Established
ID
Firm
ID
6.5
Degree
in
Ed.
Counseling;
Masters
in
EdTech
Adam
Established
ID
Firm
Course
Director
10
Degree
in
Journalism;
EdS
in
IST
Sally
Established
ID
Firm
ID
3
Masters
in
Screenwriting

9. findings
&
discussion

10. many
judgments
are
made
and
occur
in
every
part
of
the
design
process
- average
of
one
observed
judgment
every
4
minutes
- judgments
include
every
type
- framing,
appreciative,
instrumental
and
navigational
most
frequent
- default
and
off-­‐hand
judgments
next
most
frequent
- core
judgments
least
available
for
observation
- judgments
are
made
on
a
continuous
basis
throughout
projects,
and
not
just
as
mental
“adjustments”
to
models
- these
judgments
cannot
be
discovered
through
the
application
of
a
priori
scientific
models

11. judgments
were
highly
situational,
not
objective
in
the
scientific
sense
- Design
judgments
can
create
the
environment
in
which
design
activity
is
enacted,
and
reciprocally,
the
situational
qualities
of
a
particular
design
context
can
then
shape
the
kinds
of
judgments
that
can
be
made.
- design
environment/office
culture
- role
or
position
of
the
designer
- project,
client,
and
external
team
members

12. example
- Ethan’s
work
was
shaped
as
his
client
asked
for
last
minute
changes,
and
was
not
timely
in
some
of
the
deliverables;
he
exercised
his
judgment
in
interpreting
client
requests,
and
communicating
them
to
his
teammates.
- Heather
used
her
navigational
judgment
in
dealing
with
ill-­‐defined
requests
from
clients,
showing
skill
in
managing
them.
-
Gabriel
exercised
multiple
judgments
as
he
negotiated
the
details
for
an
upcoming
workshop
in
a
symposium,
as
his
client
explained
the
structure
and
vision
of
the
workshop;
the
client
valued
Gabriel’s
judgment
in
deciding
the
inclusion
of
content
- we
saw
many
design
judgments
made
in
collaboration
with
a
team
where
norms,
however
implicit,
will
always
be
assumed
to
be
in
play
–
company
guidelines
and
principles
(e.g.,
company
philosophy,
common
knowledge
base
built
up)
were
also
referenced
in
the
design
judgments
of
Julia
and
Heather

13. judgments
are
clustered
and
layered
- multiple
design
judgment
types
are
clustered
together;
they
appear
in
complex,
contextually-­‐bound
expressions
rather
than
as
pure
philosophical
forms
- some
judgments
are
foregrounded
at
any
given
time
and
others
present
in
the
background,
even
if
they
are
important
drivers
of
action
- core
judgments
are
not
conscious
and
must
be
interpreted
through
action
- judgments
are
not
always
made
by
the
instructional
designer,
but
by
others
in
the
organization
and
by
organizational
norms

14. example
- “He
also
seemed
to
be
working
with
multiple
kinds
of
judgment
simultaneously,
such
as
considering
the
possibility
that
workshop
participants
may
be
interested
in
attending
multiple
workshops
…
and
foregrounding
his
concerns
for
the
time
of
instruction,
workshop
format,
and
purpose
of
the
workshop
”
(Gabriel)
- She
started
the
meeting
[with
a
client
via
teleconferencing]
with
questions.
She
needs
clarification
on
the
document
to
understand
the
content
better
and
said
she
wanted
to
ask
questions
to
the
person
who
created
high
level
outline.
[...]
She
continued
making
clarification
on
the
understanding
of
the
content
and
she
asked
“what
communication
skills
and
active
listening
skills
mean.
She
wanted
to
sure
whether
what
she
understood
is
same
what
they
mean
with
these
terms.
Then,
she
stated
that
she
would
like
to
have
the
definition
of
“active
listening
skills.
(Claire)

15. limitations
- Data
limited
to
one
ID
at
a
time,
although
others
appear
in
the
frame
- No
continuity
across
projects
or
project
teams
- Convenient
sample,
with
most
IDs
originally
educated
in
a
single
ID
program
- In
some
instances
participants
over-­‐explained
practice
during
the
observation
- Operationalization
of
a
philosophical
model
of
judgment
is
incomplete

16. implications
for
teaching
instructional
design
This
study
is
limited
but
suggestive
–
with
a
window
into
how
design
judgment
is
exercised
in
practice
we
can
consider
ideas
like
these:
- Design
judgment
is
a
continuous
faculty
which
designers
cannot
escape
in
any
part
of
the
process
–
therefore
it
cannot
be
taught
as
if
it
were
an
individual
moment
in
design,
or
as
if
it
were
already
built
in
to
the
tools
they
are
learning
- if
judgments
cannot
be
made
outside
the
fully
complex
context
of
action
and
then
applied
to
that
context,
designers
cannot
stand
outside
a
situation
and
understand
it
(make
appreciative
or
navigational
judgments)
-­‐-­‐
therefore,
we
cannot
prepare
students
for
design
action
outside
a
fully
complex
situation
by
teaching
them
analysis
as
if
it
were
separate
from
design

17. implications
for
teaching
instructional
design
This
study
is
limited
but
suggestive
–
with
a
window
into
how
design
judgment
is
exercised
in
practice
we
can
consider
ideas
like
these:
- Design
judgments
themselves
are
both
varied
and
complex
-­‐-­‐
they
are
not
made
in
isolation
and
therefore
should
not
be
taught
in
isolation.
- With
multiple
forms
of
judgment
required
for
designing,
which
form
of
judgment
to
exercise
at
a
given
time
is
itself
a
judgment
and
a
context-­‐
dependent
one
–
there
is
no
comprehensive
means
by
which
to
anticipate
what
judgments
students
will
have
to
make
and
when
they
will
need
to
make
them;
we
need
to
help
them
recognize
when
judgments
have
to
be
made
and
help
them
develop
their
capacity
to
make
judgments
…
this
involves
personal
development
that
cannot
wait
until
they
get
“on
the
job”

18. references
- Boling,
E.,
&
Smith,
K.
M.
(2014).
Critical
issues
in
studio
pedagogy:
Beyond
the
mystique
and
down
to
business.
In
B.
Hokanson
&
A.
Gibbons
(Eds.),
Design
in
educational
technology
(pp.
37-­‐56).
Switzerland:
Springer
Verlag.
- Boling,
E.,
Hardre,
P.,
Easterling,
W.,
Korkmaz,
N.
and
Howard,
C.
(
2010).
How
do
we
Perceive
Design
Character
in
Educational
Technology"
Association
for
Educational
Communications
and
Technology,
Anaheim,
CA;
October,
2010.
- Brandt,
C.
B.,
Cennamo,
K.,
Douglas,
S.,
Vernon,
M.,
McGrath,
M.,
&
Reimer,
Y.
(2013).
A
theoretical
framework
for
the
studio
as
a
learning
environment.
International
Journal
of
Technology
and
Design
Education,
23(2),
329-­‐348.
doi:10.1007/s10798-­‐011-­‐9181-­‐5
- Christensen,
T.
K.,
&
Osguthorpe,
R.
T.
(2004).
How
do
instructional-­‐design
practitioners
make
instructional-­‐strategy
decisions?
Performance
Improvement
Quarterly,
17(3),
45-­‐65.
- Cross,
N.
(2010).
Designerly
ways
of
knowing.
Springer.
- Ertmer,
P.
A.,
York,
C.
S.,
&
Gedik,
N.
(2009).
Learning
from
the
pros:
How
experienced
designers
translate
instructional
design
models
into
practice.
Educational
Technology,
49(1),
19-­‐27.
- Holt,
J.
E.
(1997).
The
designer's
judgement.
Design
Studies,
18(1),
113-­‐123.
- Kirschner,
P.,
Carr,
C.,
Merriënboer,
J.,
&
Sloep,
P.
(2002).
How
expert
designers
design.
Performance
Improvement
Quarterly,
15(4),
86-­‐104.
- Korkmaz,
N.,
&
Boling,
E.
(2014).
Development
of
design
judgment
in
instructional
design:
Perspectives
from
instructors,
students,
and
instructional
designers.
In
Design
in
educational
technology
(pp.
161-­‐184).
Switzerland:
Springer
Verlag.
- Koszalka,
T.,
Russ-­‐Eft,
D.,
Reiser,
R
(with
Senior-­‐Canela,
F.Grabowski,
B.
&
Wallington,
C.J.)
(2013).
Instructional
design
competencies:
The
standards
(4th
Ed).
Charlotte,
NC:
Information
Age
Publishing.
- Nelson,
H.
G.,
&
Stolterman,
E.
(2012).
The
design
way:
Intentional
change
in
an
unpredictable
world
(2nd
ed.).
Cambridge,
MA:
MIT
Press.
- Polanyi,
M.
(1966).
The
tacit
dimension.
Garden
City,
NY:
Anchor
Books.
- Rowland,
G.
(1992).
What
do
instructional
designers
actually
do?
An
initial
investigation
of
expert
practice.
Performance
Improvement
Quarterly,
5(2),
65-­‐86.
- Rowland,
G.,
Fixl,
A.,
&
Yung,
K.
(1992).
Educating
the
reflective
designer.
Educational
Technology,
32(12),
36-­‐44.
- Rowley,
K.
(2005).
Inquiry
into
the
practices
of
expert
courseware
designers:
A
pragmatic
method
for
the
design
of
effective
instructional
systems.
Journal
of
Educational
Computing
Research,
33(4),
419-­‐450.
- Roytek,
M.
A.
(2010).
Enhancing
instructional
design
efficiency:
Methodologies
employed
by
instructional
designers.
British
Journal
of
Educational
Technology,
41(2),
170-­‐180.
doi:10.1111/j.1467-­‐8535.2008.00902.x
(continued)

19. references
- Schön,
D.
A.
(1987).
Educating
the
reflective
practitioner:
Toward
a
new
design
for
teaching
and
learning
in
the
professions.
San
Francisco,
CA:
Jossey-­‐Bass.
- Silber,
K.
H.
(2010).
A
principle-­‐based
model
of
instructional
design.
In
K.
H.
Silber
&
W.
R.
Foshay
(Eds.),
Handbook
of
improving
performance
in
the
workplace,
instructional
design
and
training
delivery
(Vol.
1,
pp.
23-­‐52).
San
Francisco,
CA:
Pfeiffer.
- Smith,
K.
M.
(2008).
Meanings
of
"design"
in
instructional
technology:
A
conceptual
analysis
based
on
the
field's
foundational
literature.
Dissertation.
- Smith,
K.
M.,
&
Boling,
E.
(2009).
What
do
we
make
of
design?
Design
as
a
concept
in
educational
technology.
Educational
Technology,
49(4),
3-­‐17.
- Tracey,
M.
W.,
&
Boling,
E.
(2014).
Preparing
instructional
designers:
Traditional
and
emerging
perspectives.
In
Handbook
of
research
on
educational
communications
and
technology
(pp.
653-­‐660).
New
York,
NY:
Springer
New
York.
doi:10.1007/978-­‐1-­‐4614-­‐3185-­‐5_52
- Vickers,
S.
G.
(1984).
Judgment.
In
The
vickers
papers
(pp.
230-­‐245).
London:
Harper
&
Row.
- Wedman,
J.,
&
Tessmer,
M.
(1993).
Instructional
designers
decisions
and
priorities:
A
survey
of
design
practice.
Performance
Improvement
Quarterly,
6(2),
43-­‐57.
- York,
C.
S.,
&
Ertmer,
P.
A.
(2011).
Towards
an
understanding
of
instructional
design
heuristics:
An
exploratory
Delphi
study.
Educational
Technology
Research
and
Development,
59(6),
841-­‐863.
doi:10.1007/s11423-­‐011-­‐9209-­‐2