An introduction to conducting focus groups for social science research. The course includes information on developing protocol guides, moderating focus groups, analyzing results, and reporting findings.
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Introduction to Focus Groups, Odum Institute, October 30
1. RTI International
Intro to Focus Groups
Emily Geisen
Amanda Wilmot
10/30/2013
RTI International is a trade name of Research Triangle Institute.
www.rti.org
2. RTI International
Ground Rules
Cell phones turned to vibrate or off
Refreshments and restrooms any time
Several bio-breaks throughout
Questions are encouraged at end of each section
3. RTI International
Course Outline
The role and use of focus groups
2. Focus group demonstration
3. Developing the protocol guide
4. Moderating focus groups
5. Recruiting and scheduling
6. Participant selection
7. Data management/analysis
8. Reporting findings
9. Drawing wider inferences
10. Qualitative Research Ethics
1.
4. RTI International
1. The Role and Use of Focus Groups
RTI International is a trade name of Research Triangle Institute.
www.rti.org
5. RTI International
What is a Focus Group?
“Focus group interviews typically have five characteristics
or features: (a) people, who (b) possess certain
characteristics, (c) provide data (d) of a qualitative nature
(e) in a focused discussion.”
-Focus Groups: A Practical Guide for Applied Research (Krueger)
6. RTI International
Examples
A group of coal miners sit around a small conference
room discussing mine safety. They all agree that safety
is important, but some argue that existing safety
regulations are too unrealistic or impractical to be
implemented.
A group of employees sit in a conference room
discussing access to email and other work functions
from mobile devices. One participant says answering
emails on his phone will save him time. Another
participant worries that her supervisor will expect her to
reply to emails at all hours if she has access on her
phone.
7. RTI International
What is a Focus Group, again?
Researchers have a topic they want to know about:
–
–
These topics are communicated to a group of
participants to discuss
–
–
Why aren‟t coal miners following safety precautions?
What concerns do employees have about mobile email access?
Pre-scripted questions to generate discussion
Targeted moderation to focus the discussion
Researchers summarize and interpret what they have
learned
8. RTI International
Uses of Focus Groups
Exploration and Discovery
–
–
–
Context and Depth
–
–
–
Collect info that will be used to guide development of a survey
Collect info for a needs assessment
Test a new product before development
Adding new topics or population groups to a survey
Delving deeper into a survey topic
Testing a product with a new audience
Interpretation
–
Deciding how to use survey results
–
Discuss applications with end users
-Focus Group Guidebook (Morgan)
9. RTI International
Uses of Focus Groups*, Continued
Stand-alone data collection method, or
Supplement other qualitative or quantitative
research methods
–
–
–
–
In-depth personal interviews
Survey development
Needs assessment
Product development
10. RTI International
Focus Groups for Survey Research
Identify topics/themes that are of interest to population
–
–
Get more detail on a topic so that you don‟t have to ask
open-ended questions
–
Use this to add questions on these topic/themes
Cut questions that are not relevant, not applicable, or show no
variation
Which of the following are reasons why you do not have health
insurance? Too expensive, No health problems, …
Get reaction to survey materials such as advanced
letters, envelopes, etc.
Explore methods to boost cooperation: what type of
incentives would people prefer?
11. RTI International
Strengths of Focus Groups (over other
qual./quan. methods)
If you‟re unfamiliar with a topic, you can learn a lot
quickly (often used as the first step in research)
Can be used to observe interactions or group dynamics
Generate hypotheses (that can be tested using
quantitative methods)
Cheaper/quicker than other methods
12. RTI International
Limitations of Focus Groups
Some topics are too sensitive or controversial to be
discussed in a group setting (social desirability)
Discussions can get easily sidetracked/derailed
Limited generalizability of findings/results
Cannot quantify findings or determine statistical
significance
13. RTI International
Thinking about your research…
How are you going to use focus groups?
–
–
–
Exploration and discovery? How so?
Context and depth? How so?
Interpretation? How so?
Will your focus groups be stand-alone or will you use
them with other research methods?
What are the strengths/weaknesses of using focus
groups for your research?
15. RTI International
2. Focus Group Demonstration
RTI International is a trade name of Research Triangle Institute.
www.rti.org
16. RTI International
3. Developing the Protocol Guide
RTI International is a trade name of Research Triangle Institute.
www.rti.org
17. RTI International
The What, Why, Who, How, Where
What Review research objectives
Who Determine number and type of participants
Recruit and schedule participants
Where Determine test location and equipment
Why Identify major topic areas of interest
How Script focus group questions
Develop focus group guide
18. RTI International
The What, Why, Who, How, Where
What Review research objectives
Who Determine number and type of participants
Recruit and schedule participants
Where Determine test location and equipment
Why Identify major topic areas of interest
How Script focus group questions
Develop protocol guide
19. RTI International
What’s the Focus Group About: Review
Research Objectives
Start with the overall research objectives / study goals
Identify what do you already know about the topic
Determine what new information you want to learn
Determine what information you want to explore in more
detail
Decide how are you going to use this information
20. RTI International
Review Research Objectives: Example
Step
Personal Health Records Example
Study goals:
Learn about healthcare consumers
understanding, experiences, needs, and
concerns about the ways technology can
be used within the healthcare system
What do you already
know about the topic:
Research has shown that use of electronic
PHRs can save consumers money and
improve health, yet they are not being used
extensively.
What do we want to learn
through focus groups:
Why aren‟t more consumers using
electronic PHRs to manage their health
information?
How will we use this info:
Provide recommendations for improving
PHRs and promoting awareness.
21. RTI International
Review Research Objectives: Your Turn
Step
Study goals:
What do you already know
about the topic:
What do we want to learn
through focus groups:
How will we use this info:
Your Study
22. RTI International
Why: Identify Major Discussion Topics
Once you identify what you want to learn, identify major
discussion topics (brainstorm)
Why are we doing these focus groups:
–
–
–
What is the most important question your research must answer?
What‟s the next most important question?
And so on …
Consult with project team members as needed
Consider how topic areas may need to differ by
participant selection
23. RTI International
Identify Major Discussion Topics: Example
Why aren‟t more consumers using electronic PHRs to
manage their health information?
Consumers who have not used electronic PHRs
How familiar are consumers with electronic PHRs?
How do consumers currently manage their health
information?
What are the most important factors consumers consider
when deciding how to manage their health information
What are their biggest concerns/fears about using PHRs?
Narrow list of discussion topics to about 4 major topic
questions per hour (more if questions are more specific)
24. RTI International
How: Scripting Questions
Avoid yes/no or short-answer questions, unless:
Open-ended questions generate discussion, however:
–
–
–
Open-ended follow up is planned (e.g., why/why not?)
Used to tally group (e.g., those in favor/opposed)
Keep questions focused, one-dimensional
Be careful of “Why?” questions
Use scripted probes to focus or narrow topics
Questions should be easily comprehensible: use familiar
words/terms
Questions should be reflective, not hypothetical
Move from general to more specific
25. RTI International
Scripting Questions: Example 1
Topic: How do consumers currently manage their health
information?
Bad: How do you manage your family‟s health
information?
Good: How do you currently maintain or store your and
your family‟s medical records, such as records of
vaccinations or prescriptions?
–
–
Do you keep paper or electronic records?
How does this differ for you compared to your children (or elderly
parents)?
26. RTI International
Scripting Questions: Example 2
Bad: Why do you use the mobile device you
have?
Better: What kind of mobile device do you use
at work?
Best: How did you decide what kind of mobile
device to use at work?
27. RTI International
Other aspects of Protocol Guide
Welcome/Introduction
Icebreaker
Opening (or Engagement) questions:
–
–
Exploration questions (main study questions)
Exit questions:
–
–
Easy and inviting, comfortable to discuss
Introduces the topic of discussion, provides context/background
Used to wrap-up the discussion
Check to see if there is anything else people would like to
mention that they did not get a chance to
Pre/Post Questionnaires
28. RTI International
Welcome/Introduction
Welcome
–
–
Informed consent (if required)
–
–
–
Introduce yourself, note-taker AND people behind the glass
Make participants feel comfortable
Read or summarize consent, ask participants to sign
Note audio-recording if applicable
Explain confidentiality
Go over ground rules
–
–
–
Refreshments, restrooms, cell phones off or on vibrate
No right or wrong answers
Like to hear from everyone
29. RTI International
Icebreaker
Sometimes it‟s helpful to start with an ice-breaker
Ask participant to provide their name (first name only)
Have them answer an easy question
–
–
–
Participants should be able to answer it briefly, quickly
Should not be sensitive
Can be related to focus group topic or unrelated
Example
–
–
Employee focus group: How long at company?
Department/Division?
Focus group of moms: How many children and their ages
30. RTI International
Intro Questions
Introduce Topic
–
–
Provide any needed context
–
–
–
Goal/Purpose of the study
Why participants were selected (if not obvious)
For focus group on Personal Health Records, we had to explain
what PHRs were
Show participants related materials or products
Provide background
Opening Question
–
–
Should be easy/simple to answer
Non-sensitive
31. RTI International
Pre/Post Questionnaires
Pre questionnaire:
–
–
Post questionnaire:
–
–
Use to collect information about your participants without wasting
valuable time during the focus group
Use to determine how familiar/knowledgeable participants are
with topic at start of focus group
Can be same as pre-questionnaire to monitor changes in opinion
or knowledge
Can be a handy way to summarize participants‟ opinions on
discussion topics
Keep questionnaires short (5 minutes)
32. RTI International
Putting it all together
Allow time for welcome, explaining the purpose of the
study, and consent procedures (if required)
–
Allow time for introductory or ice-breaker question
–
At least 5 minutes
Time for your main discussion questions
–
–
At least 5 minutes
About 4 question topics (per hour)
10-15 minutes per question
Exit/Wrap-up
–
At least 5 minutes
33. RTI International
Determining Question Flow
What order would you put these questions in?
1. Describe the benefits of using a mobile device for the
work you do?
2. How did you decide what kind of mobile device to use at
work?
3. What are the biggest limitations or barriers to using a
mobile device for the work you do?
4. How often do you use your mobile device for the work
you do?
34. RTI International
Question Flow
2. How did you decide what kind of mobile device to use at
work?
1. Describe the benefits of using a mobile device for the
work you do?
4. How often do you use your mobile device for the work
you do?
3. What are the biggest limitations or barriers to using a
mobile device for the work you do?
36. RTI International
Individual Practice (15 minutes)
Use your study or class example
Identify 1-2 major discussion topics for your study
Prepare an introductory question
Prepare 1-2 discussion questions with probes as needed
39. RTI International
Understanding the group Process
Forming
Testing and dependence
Dependence on the leader
Storming
Intragroup conflict
Criticism
Norming
Development of group cohesion
Optimism
Performing
Functional role relatedness
Cohesiveness
Adjourning (Mourning)
Termination, Separation
Model of group phases
(Ritchie and Lewis 2003)
40. RTI International
The Ideal Moderator:
Has adequate knowledge of topic
Has good communication skills
Is similar to / can build a rapport with participants
Can maintain balanced viewpoint
Is prepared for surprises
Reports accurately, even if it‟s uncomfortable
Has Experience with:
– group dynamics
– neutral probing/feedback
41. RTI International
What the Moderator Does
Welcome/Introductions
Leads discussion
Concludes discussion
42. RTI International
Welcome/Introduction
Make participants feel comfortable
Read or paraphrase this welcome/introduction in
protocol guide
Make sure to collect informed consent
Notify participants of observers and/or audio recording
Go over any ground rules
Let participants get refreshments and get situated before
starting actual focus group discussion
43. RTI International
Leads Discussion
Provides unobtrusive control
Manages group dynamics
Asks spontaneous probes as needed to gather more
information
Provides unbiased feedback
Summarize/verify information
Manages time
44. RTI International
Provide Unobtrusive Control
Know when to stop talking
–
Learn to focus the discussion
–
–
You said X, can you tell me more about that?
Note topics said, that you want to circle around to
Manage the flow of the discussion
–
–
Don‟t constantly interject, let participants discuss back and forth
Don‟t have to follow script, allow natural segues
If a topic you plan to cover later is brought up, indicate that you‟ll
talk more about that later.
Encourage different point of views
–
–
Has anyone had a different experience?
Does anyone view it differently?
45. RTI International
Tips for Managing Group Dynamics
Dominant talkers:
–
–
Ramblers/Off-topic:
–
–
Don‟t look at them when you ask a question.
(Politely) cut them off: “Thank you Amanda. Does anyone else
have an experience they‟d like to share?”
Avoid eye contact after so long off topic, look down or at clock
Be prepared to interject at next pause
Shy participants:
–
–
Maximize eye contact
Call by name: I‟ve heard from some of you on this and I want to
hear from the rest of you. Sarah, what is your experience with X?
46. RTI International
Tips on Spontaneous Probing
Leading probes:
–
–
–
So you had a hard time with that then?
You didn‟t want to learn more about that?
Was that difficult for you?
Neutral probing:
–
–
–
–
Can you tell me more about that?
Would you explain that further?
Can you give me an example?
Is there anything else?
47. RTI International
When you hear yourself asking a leading
question, balance it
“So you think
that‟s difficult
then?”
Leading
question
47
“...or was it
easy?”
Balanced
question
48. RTI International
Tips on Neutral Feedback
Monitor your reaction to participants/discussion
Be careful of nonverbal cues (leaning back, crossed
arms)
Limit head nodding: shows you‟re listening, but also
implies agreement
Provide quick verbal responses periodically (uh-huh,
yes, mmm, okay, go on)
Avoid loaded feedback (which implies agreement)
–
–
Loaded feedback: “That‟s good!” “Excellent.”
Neutral feedback: “That‟s helpful” “That‟s interesting.”
49. RTI International
Summarize/Verify Information
As needed, summarize/paraphrase what has been said
–
–
See if anything has been missed?
–
–
Did I miss anything?
Does anyone have anything else to add?
Probe on any subtopics you didn‟t cover
–
I understand that work-life balance is your most important
concern.
Safety is critical, but some of the precautions you‟re asked to
follow are pointless or unrealistic. Is that correct?
We talked about X, but I‟m also interested in Y
Move on to next topic
50. RTI International
Conclusion / Wrap-up
Be sure to leave time for wrap-up, especially for
sensitive topics or intense discussions
Ask participants to identify most important issues
–
–
Summarize major themes (ask note-taker for help)
–
–
Of everything we have discussed, what is the most important?
If you could make one request to X, what would it be?
Ask if its an adequate summary
Ask if anything has been missed, anything to add
Check with observers to see if they have any
questions/clarifications
Pay participants!
51. RTI International
Dealing with Difficult Situations
Too many people show up
Participants bring their children
Only a few attend
Group is reluctant to talk
Can‟t get them to stop talking
52. RTI International
Plan Your Response…
20 minutes into the focus group, and Sarah has not said
anything…
You can‟t get Joe to stop talking…
John mentions something interesting, but you don‟t want
to interrupt because there‟s a good discussion going…
53. RTI International
Role of Note-Taker
Can Handle logistics & refreshments
Collects signed informed consent (if required)
Takes careful notes
Does not participate in discussion
Can recap major themes at end of discussion (used
before wrap-up question)
Monitors recording equipment
Liaison between moderator and observers/clients
Debriefs with moderator after session
Assist with analysis and reports
54. RTI International
Role of Co-Moderator
Not required, but can be useful in some situations
Balance out strengths/weaknesses in moderator
Use to match moderator (without being obvious)
Switch leading focus group (good for long or intense
focus groups)
Support leader by keeping on track, recapping major
themes, etc.
55. RTI International
Moderating style
Different styles for different people
–
–
Spontaneous flow
Follow guide strictly
Adapt to your style, but plan ahead
–
–
Spontaneous: Be sure to note what you‟ve covered and what you
haven‟t so that you don‟t forget anything.
Strict: “That‟s a very interesting point, and we‟re going to get to
that in a minute, but right now I want to focus on X:
56. RTI International
Group Practice Moderating (30 min)
RTI International is a trade name of Research Triangle Institute.
www.rti.org
58. RTI International
5. Recruiting and Scheduling
(review on your own)
RTI International is a trade name of Research Triangle Institute.
www.rti.org
59. RTI International
At Least a Month Before Focus Group
Decide on what you‟re testing
Decide on the participants (see Section 6) :
–
–
–
Who you want to recruit
How many
How you will find them
Plan the test date
–
–
Find a date when your stakeholders can observe
Decide on your location
60. RTI International
How to Find Your Participants
Frame/list if available
Advertisements: Craigslist, flyers, newspapers
–
–
Participant databases/recruiting orgs
–
–
Pros: quick, easy, cheap
Cons: yields younger, higher educated users
Pros: quick and easy, pay per user recruited
Cons: can lead to “professional respondents”
Other sources: word of mouth; sample lists; clubs,
churches, and societies; snowball recruiting
61. RTI International
What to Pay Participants?
Are participants allowed to accept monetary incentives?
–
–
Federal employees
Participants who are being paid by their employer
Is it appropriate for your organization to pay participants?
Is topic interesting/important enough that incentives are
not necessary
How much should you pay them?
–
–
–
Trade-off between incentive amount and recruitment effort
Tailor amount to population and location
Will participants need to pay for parking?
62. RTI International
Recruiting Tips
Recruit extras due to no-shows or cancellations
–
Schedule sessions about 3-4 weeks ahead
–
–
Example: Recruit 12 to ensure at least 8 show up
Any earlier and they may forget or make new plans
Any later and you may not be able to recruit enough
participants
Send them an email or letter confirmation
Remind them the day before (by email or phone)
63. RTI International
Testing Location
At a minimum, use a large conference room (at your
organization‟s offices, hotel conference room)
For numerous focus groups: formal observation rooms
are nice (see picture on next slide)
–
–
–
One-way mirror so you can see them, but they can‟t see you
Microphones in ceiling to pick up voices, but not other sounds
(paper shuffling)
Video-taping capability
Rent a focus group facility
–
Usually very nice but can be expensive ($1,000 for 2-hour
session)
65. RTI International
Create a Virtual Observation Room using
screen-sharing software
Examples: Skype, Go to Meeting
Fosters collaboration
–
–
Improved schedule
–
–
Can accommodate observers from any location
Facilitate discussions in conference setting
Stakeholders get information immediately
No waiting for recorded videos or report
Cheaper: Inexpensive compared to travel costs
66. RTI International
Note-Taking
Recommend having a note-taker and audio or videorecording the session
Note-takers should be in the room or observation room
For audio-recording, consider getting additional
microphones so that you can hear everyone
Video-recording can be helpful, but makes some
participants uncomfortable
Participants usually forget they‟re being recorded/
observed after 5-10 minutes
67. RTI International
At Least Three Weeks Before Focus Group
Get started on recruitment
Organize your incentives
Develop your focus group guide
68. RTI International
One Week Before the Focus Group
Final your focus group guide
Organize roles in the test:
–
–
–
–
Meet and greet
Observers/Stakeholders
Moderator
Note-takers
Do a practice run on any equipment
Arrange any refreshments
–
–
–
At a minimum provide beverages
Tailor food to participants / time of day
Nothing messy or loud
69. RTI International
Prepare your materials
Develop consent forms, screeners
Instructions/directions for participants
Prepare any visual materials for participants
Pretest/posttest questionnaires
70. RTI International
The Day Before / Morning Of
Send out reminders:
–
–
Phone or email to respondents
Email to observers, stakeholders, note-takers
Equipment/Facility
–
–
–
–
Make sure the room you‟ll use is tidy
Make sure your meet/greet person
has the final list of participants‟ names
Incentives are available
Check any equipment (video/audio recording)
72. RTI International
Purpose of the session
To understand the principles of participant selection for
focus group research
To consider the practicalities of participant selection for
focus group research
73. RTI International
Focus groups: a qualitative data collection method
Focus groups are a qualitative data collection method
When reporting the findings the rationale should be
provided for the research along with the methods used
Is the research…
–
–
–
–
Justified
Rigorous
Systematic
Transparent
74. RTI International
Sampling for qualitative research
The design of a sampling strategy for qualitative
research is as important as that for quantitative research
Qualitative research uses non-probability sampling
75. RTI International
Informing the design of a qualitative sampling strategy
What are the research objectives?
What is the scope of the research?
Who is out of scope and should be excluded from the sample?
Who is in scope and should be included in the sample?
What is the budget?
What is the reporting time period?
What sampling technique will be employed?
How are the data to be analyzed?
What data collection methods should be employed?
What are the sample criteria?
What size should the sample be?
What will be used as the sampling frame?
How are potential respondents/participants recruited?
76. RTI International
Sampling technique
Quantitative sampling
Probability sampling
Members of the research population are chosen at random and have a
known probability of selection
The aim is to produce a statistically representative sample
Qualitative sampling
Purposive non-probability sampling
The number of people interviewed is less important that the criteria used to
select them
Members of the research population are chosen on the basis of their
characteristics to reflect breadth and diversity of the research population
We do not aim to produce a statistically representative sample or draw
statistical inference
77. RTI International
Sample criteria
What characteristics will need to be reflected in the
sample population to ensure breadth and diversity?
Criteria used may be based on demographic
characteristics or behaviours or attitudes
Some criteria may be considered more important than
others
78. RTI International
Sample size
Small sample sizes for qualitative research
There is no need for scale because there is no need for
statistical inference
Sample size determinants:
Heterogeneous or homogenous nature of sample population
Number of selected criterion
Scale
10 to 50 for one-to-one investigation
40 to 100 for group interview
79. RTI International
Sampling frames
A sampling frame is a list that identifies units within the
target population
Frame evaluation:
Comprehensive
Sufficient numbers
Geographical dispersion
Respondent contact details correct
82. RTI International
Study about attitudes and behaviors surrounding
dental attendance
Selection criteria initially considered
Age to ensure demographic balance
Gender because patterns of attendance differ between men and women
Family Unit composition because attendance by others in the family might
influence the respondents attitudes or behaviours
Employment activity because attendance might be affected by time constraints
during working hours
Income as a known factor affecting dental attendance patterns
Regional location as dental attendance varies across the country
Ethnic origin as may influence attitudes or behaviours
Type of area as urban/rural location may affect attitudes and behaviours
Dental health to explore how attitudes vary among people with different dental
health
Current pattern of dental attendance (regular/irregular/occasional) for comparative
analysis
Illustration from Ritchie J and Lewis J „Qualitative Research Practice‟ (2006)
83. RTI International
Prioritised selection criteria
Primary criteria
Dental attendance pattern
Age
Gender
Region
Family Unit
Employment activity
Income
Secondary criteria
Dental Health
Ethnic Origin
Type of Area
Illustration from Ritchie J and Lewis J „Qualitative Research Practice‟ (2006)
84. RTI International
Sample Matrix
Area 1 of 6
Group 1
18-29
Group 2
30-44
Group 3
45+
4 men
1-2 regular attenders
1-2 irregular attenders
1 occasional attender
4 men
1 regular attender
1-2 irregular attenders
1-2 occasional attenders
4 men
1-2 regular attenders
1 irregular attender
1-2 occasional attenders
4 women
1-2 regular attenders
1-2 irregular attenders
1 occasional attender
4 women
1 regular attender
1-2 irregular attenders
1-2 occasional attenders
4 women
1-2 regular attenders
1 irregular attender
1-2 occasional attenders
Spread of family unit type and employment activity
Illustration from Ritchie J and Lewis J „Qualitative Research Practice‟ (2006)
87. RTI International
7. Data Management and analysis
RTI International is a trade name of Research Triangle Institute.
www.rti.org
88. RTI International
Purpose of the session
To understand the principles of good qualitative focus
group analysis
To consider the practicalities of analysing qualitative
data
89. RTI International
Acknowledgement
Some of the slides contributed by the UK National
Centre for Social Research (NatCen).
The Framework methodology was developed by
Jane Ritchie and Liz Spencer in the 1980s at NatCen.
http://www.natcen.ac.uk/events-and-training/ourtraining
91. RTI International
Transcription
Transcription is time consuming
The focus group transcription should capture the
discussion in it‟s entirety
Focus groups are more difficult to transcribe than one-toone interviews as there are multiple participants to
distinguish between
It may be important for the analysis for the transcriber to
be able to identify each participant‟s words. The
transcriber will use the notetaker‟s notes to help
determine who is speaking
Check on quality of the transcription
Anonymize transcript
93. RTI International
Nature of qualitative analysis
Quantitative research
Qualitative research
–
- Creating categories
–
–
Fixed categories aiming to
enumerate
Requires enumeration in
order to be meaningful
Test hypotheses and
provide explanation
- Developing explanations
- Does not need enumerating
- Range and diversity key
94. RTI International
Aim of analysis - analytical outputs
Categories of things (thematic analysis)
Reasons for gambling
Sources of debt advice
Categories of people or processes (typologies)
Types of parents of children with learning difficulties
Types of welfare benefit claimants
Explanations of attitudes, choices or impacts
(explanatory analysis)
Factors influencing how people save for retirement
What helps achieve positive outcomes from an employment
programme
95. RTI International
The nature of qualitative data
The data are:
Voluminous
Unwieldy
Multi-stranded with overlapping categories and theme
106. RTI International
Frame construction: individual data
One chart represents one main theme
Chart each individual case
–
–
–
Each individual as one row
Subthemes in each column
Give cases same space on each thematic chart
107. RTI International
Frame construction: group data
Chart as a whole group
–
–
One group per page, tracing individual cases
–
–
–
Each group as one case
Several groups on each thematic chart
Order cases in the same way
Give cases same space on each thematic chart
Don‟t be alarmed if every cell is not filled for each case
Guidelines for entering data
–
–
May be useful to keep a record of group process
Levels of participation, context in which comments made, how
views evolve/develop/change
109. RTI International
Stage 3: Testing the analytic framework
Testing framework is important
–
multi-stranded & overlapping categories
– different experiences
Test by coding or indexing
–
apply numerical code to categories and sub-categories
– code using software or write codes in margin of
transcript
Test by piloting
–
enter data from small number of transcripts
Checking for: overlap / gaps
110. RTI International
Data management coding / indexing
Example of analytical framework index for sexual identity project
1. Demographic details
2. Defining sexual identity
2.1 Conceptualization
2.2 Salience
2.3 Self categorization
2.4 Changes over time
3. Language used
3.1 Terminology used
3.2 Categories used
4. Acceptability
4.1 Acceptance by society
4.2 Understanding purpose of questioning
4.3 Trust in data security
4.4 Trust in data collector
5. Other
111. RTI International
Developing an analytical framework exercise
Groups to discuss analytic framework that could be used
for the morning focus group session
Create main themes and subthemes
Post-it notes
15 mins
Feedback – group spokesperson
123. RTI International
Categorization and classification
Classification of sexual identity
Relevant dimensions
–
–
–
–
Conceptualization
Salience
Self-categorization
Changes over time
Typology
–
–
–
Latent identifiers
Conscious identifiers
Reluctant identifiers
(2)
124. RTI International
Associations and explanations
Linkages between:
–
–
–
Two attitudes
Attitude and behavior
Circumstances and need
Verified through explanation
–
–
Explicit respondent accounts
Implicit identified by the researcher
125. RTI International
Making explanations
Informed by
–
–
–
Involves
–
–
–
Hunches and hypothesis
Reflections during field work and analysis
Other research or theories
Detailed within case analysis
Comparison between cases
Repeated interrogation of data
Comprehensive
Expect multiplicity
126. RTI International
Levels of classification and interpretation
In summary:
Descriptive categories
–
Factors, reasons, impacts
Classifications or typologies
Explained associations
Meaning
Theory/Strategy generation
128. RTI International
8. Reporting the Findings
RTI International is a trade name of Research Triangle Institute.
www.rti.org
129. RTI International
Purpose of session
To provide a basic overview of the principles involved in
reporting qualitative findings and constructing a final
report.
130. RTI International
Aims of the final report
To complete and document the research process
To inform key stakeholders of the findings
To communicate the findings in a clear and coherent
way
To guide readers in the interpretation of findings (avoid
misinterpretations)
131. RTI International
Basic principles for writing
Tell the story
Structure the report
Know your reader
Write in plain English
Use visual representation
132. RTI International
Reporting the Findings
Revisit the objectives (ensure the report covers these)
Consider the audience
Check report format
Consider time available
Consider length of the report (focus on relevant points
only)
Report findings in past tense
133. RTI International
Structuring the report
Title
Contents
Executive summary
Introduction / Background
Methodology
Findings
Conclusions
Recommendations
Appendices e.g. sample matrix, interview guide
135. RTI International
Defining the boundaries
Research rationale
Methodology
– Sampling strategy
– Data collection
– Method of analysis
– What can or cannot be inferred form the findings
Examples of relevant documentation
– Interview guide
– Contact letter
– Screening questionnaire
– Sample matrix
Substantiate conclusions by grounding the data in the findings
– Examples
– Quotations
– Case illustrations
136. RTI International
Using Quotations
Avoid over-use
Avoid under-use
Present range and balance
Edit sparingly
Should provide illustration only
Amplify but do not repeat a point
Avoid reliance on a few articulate respondents
Avoid very long quotations
Identify relevant characteristics
Preserve participant confidentiality
137. RTI International
Quotation examples
“I don‟t feel you have to be out in all contexts…” [Gay
man, 35-44]
“The Government uses statistics to show what it wants to
show, for example unemployment, they‟ve used all sorts
of different measures for employment to show it‟s going
down…rather than it necessarily going down itself.”
[MHE Wales]
“They don‟t XXX care about us. It‟s everyone for
themselves” [Male, 18-24, Central London]
138. RTI International
Quotation examples
“I think it depends quite a lot on the way you live and
how old you are. For example, I was born down [South
West county], and my parents being extremely
religious, have very definite views on sexuality.”
[Bisexual woman, aged 35-44]
“I‟ve lived in [Muslim country], nobody would ever admit
that [being gay], so they all choose to be heterosexual.
Somebody‟s not been here that long, it‟s even more of
an issue.” [Heterosexual woman, aged 35-44]
139. RTI International
Is a quotation necessary?
The actual presence of the question was considered to
have an important purpose in its own right. It was
thought that as the questioning became more
commonplace it would demonstrate to the wider public
that non-heterosexuality was unremarkable.
“It‟s having it on there, it‟s beginning to usualise the
question, it‟s a cultural shift, it‟s enabling people to
gradually recognise it‟s no bloody big deal. But if we
don‟t have it there, we are setting up this whole process
of we must be quiet about these people and all the rest
of it.” [Gay/lesbian woman, 55+]
140. RTI International
Avoiding quantitative language
Avoid tendency to use numbers
Do not discuss qualitative results in terms of proportions,
percentages or statistics
Indicate strength of finding by discussing in terms of:
–
A common finding, A recurring problem, Problems were
observed…. etc
Use diagrams for illustration
141. RTI International
Principles to remember
Present balanced findings
Present grounded conclusions (avoid assumptions)
Distinguish between findings and your own conclusions
Provide commentary alongside the analysis (do not
leave the reader to do this for you)
Support with evidence from other research findings (if
appropriate)
142. RTI International
Editing and reviewing
Allow some time before editing (if possible)
Review the content
Check accuracy of statements
Peer Review
Proof read
147. RTI International
Generalization
Can qualitative findings be generalized beyond the
sample and context of the study research?
Different definitions of „generalization‟.
–
Representational generalization.
148. RTI International
Representational generalization
Meaning
Inference to parent or sampled population
Whether range and diversity (experiences, explanations
etc) can be matched with the sampled population
Whether list (of experiences, explanations etc) is
inclusive of what would be found in the sampled
population
Basis
Depends on validity and reliability of findings
149. RTI International
Validity
Validity: Accuracy of representation
Has the researcher understood the issues
from the respondent‟s perspective?
Have the issues been fully articulated and explained
Are interpretations underpinned by the data
Alternative terms: Credibility and Plausibility
150. RTI International
Reliability
Reliability: Replicability
Internal reliability
–
Extent to which assessments and judgments are replicated
between researchers
External reliability
–
Extent to which findings would be replicated
if the study were repeated with the same
or different sample
Alternative terms:
Confirmability, Consistency, Dependability
151. RTI International
Drawing wider inference
Depends on:
Quality of sampling
–
Reflects diversity of sample
population
–
Includes all key constituents
–
Includes outliers and a-typical
cases
Quality of data collection
–
Free from interference: neutral
and objective
–
Probing of meaning
–
Exploration of all relevant
explanations
Quality of analysis
–
Systematic, comprehensive and
inclusive
–
Within and between case
analysis
–
Displays diversity
Quality of interpretation and reporting
–
Comprehensive
–
Clear
–
Underpinned by data
–
Displays multiplicity of accounts
and explanations
–
Non-quantitative
Documentation
–
Transparency
–
Research process
152. RTI International
Validation of the inference
Validation
– Comparison and check of fit
– Deviant cases analysis
– Member or respondent validation
– Triangulation
153. RTI International
In summary
Validity: do the data and findings accurately reflect the
behavior , views etc of respondents?
–
–
Content validity: the respondent‟s expression or articulation of the
issue
Validity of interpretation: the researchers interpretation of the
issue
Reliability: would the data and findings be replicated if
the study were repeated?
Representational generalization: are the data and
findings applicable to the wider population from which
the sample is drawn?
156. RTI International
Qualitative Research Ethics
Check the legal regulations for the State / Country
Check your organisations Institutional Review Board
(IRB) policies
Understanding “informed consent” (verbal/written)
Understanding “confidentiality”
Basic principles of good social research
–
–
Participant safety (Physical/Emotional/Psychological)
Researcher safety
Recompense
158. RTI International
Reading List
Quality in Qualitative Evaluation: A Framework for assessing research
evidence. A quality framework. A report produced on behalf of the Cabinet
Office by Liz Spencer, Jane Ritchie, Jane Lewis and Lucy Dillon, National
Centre for Social Research. ISBN: 07715 04465 8. August 2003.
Government Chief Social Researcher‟s Office Crown Copyright 2003.
Assessing the Quality of Qualitative Research. Patient Education and
Counseling 90 (2013) 1–3.
Qualitative Research Practice: A guide for Social Science Students and
Researchers. Edited by Jane Ritchie and Jane Lewis. SAGE 2003.
Focus Groups: Theory and Practice. D. W. Stewart and P.N. Shamdasani.
Applied Social Research Methods, Volume 20.
Focus Groups as Qualitative Research. David Morgan. Sage. 1989.
Focus Groups: A Practical Guide for Applied Research. Richard Krueger.
Sage. 1989.
What Are Qualitative Research Ethics? Rose Wiles. Bloomsbury Academic.
2013.
Thinking back to the description on the first slide, what characteristics do the people in the first group have in common? Right, they’re coal miners. In the second group they are employees. What kind of data are they providing? Miners are providing data about mine safety. The employees are discussing the pros/cons of getting mobile access to email. The data we’re going to get from these discussions will obviously be qualitative – based on people’s discussions. We’ll talk more about creating a “focused discussion” in Section 5.We’ll break this down a little more on the following slide.
This is another way of looking at what a focus group is from more of a research perspective.
In the quote from Krueger on the first slide, I mentioned that the focus group data is qualitative. I want to talk about what that means for focus groups.Exploration and discovery – learn about topics that are poorly understood. In the example on the previous slides, the researchers considering launching a service that allowed employees to access email on their phones (previously only available from a computer). Are employees even interested in this? What are their initial reactions/impressions?Context and depth – In the other example, the researchers knew that miners often did not follow safety precautions. They wanted to know why? Certain regulations were impractical – requiring employees to where a harness even when only three feet off the ground. Always short-handed and can’t always train someone fully before putting them on the job. Safety regulations slow them down, making it harder to do their job. They weren’t aware of certain regulations.Interpretation – We summarized what the miners said during their focus groups and then held focus groups with mine operators. Do they agree with the miners’ assessments on safety? What do they think they can do to address this concerns?A single focus group may be used for all three of these purposes
Generally, stand-alone data collection does not mean that it CANNOT be used with other methods, but that the information collected and final report are self-sufficient and can stand on their own. Maybe the results are used to decide whether to pursue a certain opportunity or not.When used with other data collection methods, sometimes its sequential: focus groups, in-depth interviews, survey development. Other times its combined with other methods of a way of “triangulating” or understanding a concept from different perspectives.
Allow a few minutes for participants to think about their research. Call on one or two people to discuss their study.
Once you finish these steps, then you’re ready to actually conduct the focus group.
We’ll cover the Who and Where later. In this section, we’re going to focus on the What, Why, and How
Although I usually think of my topics areas in question format, don’t worry too much about specific question wording. The idea is just to list all of the things you want to know. Major topic areas should be open-ended and some-what broad. Avoid yes/no questions
One of our research objectives was to learn why more consumers weren’t using electronic PHRs to manage their health information.For this particular group, we’re specifically interested in people who don’t use e-PHRs. We would likely want to do another focus group with people who did.Now we can identify the major things we want to know from these people: Have they even heard of electronic PHRs? What do they use now? Why’d they choose that? What are their biggest concerns/fears about using PHRs?
Avoid “Why?” questions: There is some consensus among the experts that “why” questions don’t work well for focus groups and there are two primary reasons. The first reason is that why questions can make people defensive. If I ask you “Why did you do that?” It can sound like I want you to justify or explain yourself. We don’t want to make our participants defensive.The second reason is that why questions are often too broad. For example: Why did you come to this focus group course? Answers might range from “My boss told me to” to “I was free this day.” But what I really want to know is: What factors did you consider when deciding whether to attend this short course?
Scripting questions is a little bit more than just taking your discussion topic and framing it in the second person instead of third person. In this example, there are two concepts that are pretty broad: manage and health information. For manage, what we really want to know is how do they keep their medical records. Do they have a file in their desk where they have stored this information? Do they just look it up in their email? Do they call their doctor and ask for records? You see that I added a few scripted probes too. One of the things we’re really interested in is paper vs electronic. This will likely come up during conversation, but if not I want to probe on it. You’ll also notice that this is one of those “short-answer” questions I told you to avoid. This is okay for a probe because it’s just for clarification purposes. However, I would likely follow up with they said with a spontaneous probe aimed to elicit more information, such as “Can you provide an example?” We’ll discuss spontaneous probes under moderating. Another probe I have is asking if this differs for themselves versus children (or other dependents). People tend to be much better about tracking things for their children then they are for themselves.
Generally, you’re going to have three types or sets of questions. Introduction questions – these are sometimes called engagement questions, but really all of your questions should be engaging. The point is that these are the questions that ease people into the focus group discussion. You want to introduce the topic and provide the necessary context and background so that people know how to answer. The questions should be easy to answer, non-threatening.Then you move into the meat of the focus group with your exploration questions.Finally, you’ll want to prepare an exit or wrap-up question. You don’t want the session to end abruptly in the middle of a heated discussion. Allow time to wind down and let participants cover any topics you didn’t ask them about. Usually I spend about a minute paraphrasing what we’ve already discussed and then ask if there is anything else participants would like to say? It helps to summarize first or else you have people telling you things they already told you.
If you’re planning an hour focus group, you can see that 15 minutes is consumed just by the welcome, intro questions, and wrap-up. That only leaves 45 minutes for your main questionsSo 4 question topics per hour may seem really short. Keep in mind that these are not necessarily scripted questions, but question topics. Depending on how broad or focused your topics are, you might have a lot more scripted probes to help generate discussion.
I think it makes sense to start with what kind of device they use as that’s going to be relevant for the rest of the questions. Plus, this is a pretty easy question to start with and could really serve as your introductory or warm-up question. Next, we could have gone with benefits or limitations. These are both more general questions then the “how often?” I went with benefits because I think it’s helpful to make people think of all the ways they use something before asking about its limitations. The “How often?” question fits with the benefits question because they are both about using mobile devices. Benefits is more general, so I put that one first. That leaves limitations/barriers as the last question.
Discuss here whether it’s good to use a project person or an outsider
Know when to stop talking: As moderator, your job is to guide discussion. It’s not an interrogation. Back and forth discussion among participants is great. Focus the discussion: Pick up on something a participant said and ask them to expand. Shows you’re listening, and you’re interested in what the participants have to sayIf a participant says something interesting, but you don’t want to interrupt, just jot it down and then come back to it: Earlier Joe mentioned he was concerned about privacy. What are you biggest privacy concerns with X?Manage the flow of discussion:Beginners often like to follow the guide rigidly, but don’t worry about that. If the discussion naturally segues from topic 1 to topic 3, go with it. You can always circle back around to topic 2.“I’m glad you brought up privacy and security. We’ll talk about that in a few minutes. Right now, I’m really interested in …”
I’ve heard from some of you on this and I want to hear from the rest of you. Sarah, what is your experience with X?Talk about duds
If someone says “I agree,” you might ask “what experiences have you had that make you feel that way?”
Periodically, and at least at the end of each major topic, I like to make sure I’m on the same page as my participants. Recap what was said – you might be surprised that you misinterpreted someone’s main point or missed the nuance of what they were sayingSee if anything was missed – from the participant’s perspectiveSee if anything was missed – from your perspectiveMove on to the next topic
Too many people show up: perhaps you recruited 12 hoping 8 would show up, but all 12 show up. However you’re conference room only fits 8. You’ll need to be prepared to dismiss some people, but you still have to pay them.Participants bring their friend or sibling. Be firm and tell them that you only have space for so many people in the group. The friend is welcome to wait for them in lobby..ChildrenIf you’re doing a focus group on stay-at-home mothers at 2pm on a Wednesday, don’t be surprised if moms bring their kids. Think about these things and provide instructions not to bring kids.If child is old enough, ask parent if child can wait in lobbyIf child is young, encourage parent to set up them up away from the table, with something to keep them entertainedOnly a few attendContinue with the focus group, allow participants more time with responses, ensure everyone responds to every questionCheck your recruiting strategies for next roundReluctant to talkMaybe your content is too sensitive, boring, participants aren’t knowledgeable about subject – consider revising for next groupEncourage them to drink coffee, sodas, eat cookiesUse more ice breakersCall on them by nameCan’t get them to stop talkingFocus group experience can be cathartic for someIf have extra time: Alert participants that the session is over, but you can stay a little longer if anyone wants to talkIf don’t have extra time: Tell participants that the session time is over and you want to be respectful of their time. Encourage participants to keep discussions going, to email or contact you, etc.Start wrapping up earlier next time
You can’t plan for everything, but it does help to go into the focus group prepared for a few of the most common situations you might discover.
Talk about co moderatorDrawing the table.
Strengths/weaknesses: If you’re doing a focus group on a very technical topic, it may help if one moderator is the substantive expert and the other is the focus group expert.Matched: There are times when you want to have a matched moderator, but you don’t want it to be totally obvious. The comoderator can either be an equal where you switch off leading or you can designate one as the primary and one as the secondaryHaving a comoderator is great for introducing a new moderator. It let’s them see a focus group in action, but they’re also required to follow along and keep up.
When I do focus groups, I like to be a little spontaneous. I like to follow the topics that people seem interested in even if it doesn’t match the order of the protocol guide I so carefully crafted. Letting people talk about what they want to talk about (if it’s also of interest to me) can help generate discussion. However, some people can get easily frazzled by jumping around and may have a difficult time keeping the focus group on track.It’s also perfectly okay to follow your guideline strictly. It actually makes it a lot easier to write up your notes. If you notice that the flow isn’t work, you can always rework the guide between focus groups.Either way, be prepared. If you’re using the spontaneous approach, it can be easy to forget what you’ve covered. You either have to be good about keeping that in your head or note on your protocol what topics you’ve already completed so that you don’t miss anything.If you want to follow the protocol strictly, think about how you’ll handle situations where people bring up topics that you’re going to talk about later.
Amanda is going to talk about participant selection after lunch.
Are participants allowed to accept incentives?Some federal employees cannot accept incentives if their participation is related to their job. Alternately if you’re doing internal focus groups and employees are being paid for their time, they might not be allowed additional incentivesIs it appropriate for your organization to pay participants?If your organization is a charity, non-profit or even a government, how’s it going to look that you’re shelling out so much money for an incentive? Is my tax money paying for this?Some topics are interesting/important enough that people don’t want money. They are just glad to help, hope their input can help others.How much should you pay them? Around here? I would say $50/hour and $75/2 hours.Trade-off between incentive amount and recruitment effortHigher incentives = less labor hours for recruitingHigher incentives = fewer no-shows/cancellationsTailor amount to population and locationProfessionals (doctors, lawyers, etc) need moreFocus Groups held in big cities (NY, Chicago) need morePay for parking?RTI has an office in Chicago, but it’s awful for conducting focus groups because it’s downtown and it can cost someone $20-30 just to park for two hours.
At a minimum, you just need a large conference room. This can be at your company’s offices, a hotel conference room, or at a focus group facility.If you plan on doing a lot of Focus Groups, formal observation rooms are certainly nice because they remove a lot of the set up time. This is a photo of the Observation room at RTI. Between the two rooms is a one-way mirror that allows observers to look out without being seen by participants. This set-up is hard-wired for video and audio recording so we don’t need speakerphones or screen-sharing software. In the observation room, is a monitor that displays whatever the participant is viewing on the computer in the other room. There are microphones in the ceiling of the test room that pick up voices, but don’t pick up sounds like typing or paper shuffling. Those are projected in the observation room through speakers. Finally, having a large conference table allows you to move from observation mode to discussion mode as needed.If you rent a focus group facility, it will have a similar sort of set up. Generally focus group facilities can be really expensive to rent – at least $1,000 for one session and that does not include food, etc.
At a minimum, you just need a large conference room. This can be at your company’s offices, a hotel conference room, or at a focus group facility.If you plan on doing a lot of Focus Groups, formal observation rooms are certainly nice because they remove a lot of the set up time. This is a photo of the Observation room at RTI. Between the two rooms is a one-way mirror that allows observers to look out without being seen by participants. This set-up is hard-wired for video and audio recording so we don’t need speakerphones or screen-sharing software. In the observation room, is a monitor that displays whatever the participant is viewing on the computer in the other room. There are microphones in the ceiling of the test room that pick up voices, but don’t pick up sounds like typing or paper shuffling. Those are projected in the observation room through speakers. Finally, having a large conference table allows you to move from observation mode to discussion mode as needed.If you rent a focus group facility, it will have a similar sort of set up. Generally focus group facilities can be really expensive to rent – at least $1,000 for one session and that does not include food, etc.
If you don’t have the formal observation room and cannot afford to rent a focus group facility, you can usually create one virtually (if you have internet access). Set up a laptop with a webcam (or nicer camera) in the corner of the room. Using Skype, GoToMeeting or other screen-sharing software to share the image with stakeholders. Rather than recording sound on the laptop, I recommend setting up a conference phone in the middle of the table, so that it’s easier to pick up all voices.I don’t recommend having observers in the testing room, primarily because it is hard from observers to refrain from interjecting. Then the respondent does not know if they should be directing their comments to you, the moderator, or the observer. That’s why screen-sharing is so useful.
I like having an in-person note-taker taking notes during the interview. They can use the recordings to fill in any gaps. I find that this is better than relying on tape transcriptions because the transcripts cannot fully capture the energy in the room, sarcasm, head-nodding, and other non-verbal cues. Plus, transcripts are expensive and you have to wait a week or two for them.
Tailor food to participants – for the coal miners, we bought them dinner. These men work hard every day and a fruit and cookie platter would not have cut it.Generally I would recommend foods that people can eat with their hands and are not too messy. Sandwich halves, non-greasy pizza, snacksDon’t get anything that is really loud as people eat it, like potato chips. It makes it hard to hear and messes with your recordings.