Presentation given at the two day NERCOMP Assessment into Action SIG, March 2009

1. Pursuing Your Research Question
With Qualitative Methods
Gail Matthews-DeNatale
Simmons College

2. You are here … but
where to begin? What
do you do now?

3. A Word on “Qualitative”
Core Beliefs
• The client/learner perspective
is different from our own
• Need to understand the life
patterns of those whom we
serve/teach
• It’s worth taking the time to
talk with, observe, and try to
“walk the life” of these people
• If we involve clients/learners
prior to (and during) the
development process, the
end result is improved
Related Terms
(What’s in a Name?)
• Ethnographic Research
• Participatory Design
• Contextual Inquiry
• Action Research

4. Qualitative Strategies: Usual Suspects
• Open-ended survey questions
• Focus groups
• Observation
• Interviews

5. Example: Simmons SSW ePortfolios
Focus group with
pilot student
participants
How would you describe an
ePortfolio to someone who
has never seen one?
What was easiest about the
experience … what was
most challenging?
With whom have you shared
your ePortfolio?
What advice would you give to
an incoming student who
has never created an
ePortfolio?

6. Example: Brighton U Usability Lab
People often forget
the steps they took,
where they got lost.
Ask participants to do
certain tasks (e.g.,
find something on a
web site), then
observe them in
action.
Note:
Lab is desirable, but not essential.

7. Example: Simmons Blended Learning

8. Example: Simmons Blended Learning
Online Journal Writing Prompts
• What did you learn about yourself as a teacher this week?
What aspects of your course would you describe as a
success and why?
• What challenges are you experiencing as a teacher in this
course? What are your quandaries or puzzlements?
• What appear to be the learning accomplishments and
challenges that your students have experienced so far in
the course?
• Looking at how you designed and deployed your course,
what things are you glad that you took the time to do?
What things would you do differently next time?

9. Example: Simmons Blended Learning

10. Thinking “Out of the Box”
• Photo elicitation (participant-created)
• Role Play / Participant Observation
• Relational Mapping
• Paper Prototyping
• Shadowing

11. Photo Elicitation
Use the camera provided to take pictures of
• Your favorite place to study
• The computer you use
in the library
• One picture of the libraries
to show to a freshman
• All the stuff you take to class
• The place you keep your books
• Your favorite person or people to study with
• A place in the library where you feel lost
• The night before a big assignment is due
Source: Foster and Gibbons, Studying Students (2007)

12. Role Play
“What about designing
something that isn’t physical,
something like a service or
experience, interactions over
time? The best way to get a
feeling for flaws in your design
is to act it out. A place where I
think the effort is really worth
while, is where people are
wrestling with quite serious
problems: things like education,
security, finance, or health.”
Tim Brown (CEO of IDEO)

13. Participant Observation
“Another way to [deepen
understanding] is to put
ourselves through an
experience that we are
designing for. To project
ourselves into an experience.”
Tim Brown (CEO of IDEO)

14. Relational Mapping

15. Paper Prototyping

16. Shadowing

17. 15 Minute Breakout (Record on Flipchart)
• Which methods are relevant to your group’s
research? What other strategies come to mind?
• How might you use several methods in
conjunction with one another to gather a rich,
triangulated, set of data?
• Of the methods you deem relevant, which ones
could you implement without additional support or
training?
• What additional resources will you need, and who
on your campus might be able to provide that
support/training?

18. Interviewing: Asking Good Questions
"I've got two pieces of bad news about that
experimental English comp course where students
used computer conferencing.
First, over the course of the semester, the
experimental group showed no progress in abilities
to compose an essay.
The second piece of bad news is that the control
group, taught by traditional methods, showed no
progress either."
From Steve Ehrman, “Asking the Right Questions”

19. Interviewing Truisms
No such thing as a “perfect” interview
Record practice interviews, review and critique with
a trusted friend, person with expertise, etc.
The better you listen, the more articulate your
interviewees will become
Practicing with recording equipment beforehand is
like carrying an umbrella
Your last interview will be orders of magnitude better
than your first – plan accordingly

20. Sequence Matters
Begin with “easy” low stakes questions to develop
comfort and rapport
Cluster questions according to themes, but only ask
one question at a time
Question clusters should begin with a general query,
each question becomes increasingly specific
Don’t go lock-step through your questions
Consider using pre-interview exercises (e.g.,
mapping) to “prime the pump”

21. Wording and Pacing Matters
Whenever possible, use open-ended questions
(questions for which there isn’t a yes/no answer)
Beware of questions that betray your assumptions
(As a math major, in what ways are your needs for
the writing lab diminished?)
Good follow-up questions are the mark of a pro
(for example? … in what way? … when was that?)
Silence is a strategy – if you wait and appear to be
interested, most interviewees will elaborate

22. Individually, take five minutes to jot down interview
questions that would be relevant to your group’s
project.
As a group, pool questions – what patterns or
clusters emerge?
Select 5-10 questions that could be used in a one-
on-one interviews for your project. Please also note
who you would be interviewing (e.g., students in x
year or y department, adjunct faculty, pre-tenure
faculty, post-tenure faculty, etc.)
15 Minute Breakout (Record on Flipchart)

23. Gail Matthews-DeNatale
Simmons College
gmdenatale@simmons.edu