2. Methodology and IRB/URR
Dr. James Lani
Take Away Message
Research design is a blueprint with
several components:
⢠Research method: Quant/Qual/Mixed
⢠Operational constructs: How
constructs are measured
⢠Sampling strategy and procedure
What is Research
Research follows the scientific method.
Whatâs the scientific method?
⢠Formulation of testable questions
or hypotheses
⢠Itâs organized knowledge: logical
(theory) and evidence based
(observable)
⢠Precise constructs
⢠Can be disproven (falsifiable)
⢠Parsimonious (simplest
explanation)
3. Methodology: The Cookbook Metaphor
Cooking Researching
Making a stew⌠Examining research questionsâŚ
Ingredients: 2 lbs beef, I clove garlicâŚ
Preparation: Cube beef, mince garlicâŚ
Cooking Instructions: Bake at 350 for one
hour
Celebrate: Youâve made a replicable) stew!
Ingredients: 20 participants, 15 item
questionnaire, semi-structured interview.
Preparation: Administer questionnaire before
and after lecture; semi-structure interview with
participants for 20 minutes using a tape
recorder.
Data analysis plan: Conduct dependent sample
t-test; transcribe interviews then thematize
participants responses.
Celebrate: Youâve conducted a (replicable)
research study!
Where the recipe can be replicated
5. Methods Quantitative Qualitative Mixed
You can count it Not quantitative Both
Goal
Tend to answer âWhat
questionsâ (What is
relationship or
differences) or âWhen
questionsâ (when is
theory supportedâŚafter
intervention
Tend to answer âWhy
questionsâ (understand
why people feel that way)
or âHow questionsâ
(explore how they see
things)
Both
Research methods/
Strategies of data
collection
Experimental (random
assignment) and Non-
experimental methods
(no control group),
Observations (time
participant)âŚ
Semi-structured
interviews, Archive data,
Observations (write down
positive and negative
feeling words)⌠Both
Operationalize
variables
Define constructs/
instruments
Define constructs/
instruments Both
Sampling strategy Discuss sampling process Discuss sampling process
Both
6. Theoretical (or Conceptual) Framework
Theory is a systematic explanation of behavior
of phenomena.
⢠Theory guides analyses
⢠List existing theories and how your
research questions relate to those
theories
⢠E.g., Theory of mind: the ability to understand that
others have their own beliefs, desires, intentions.
Empathy. Tested by Faux Pas Task (ability to
recognize a faux pas). Research question: Does
alcohol abuse impact empathy?
7. Population
The population is the group you want to
generalize to.
⢠Describe characteristics of population
⢠Why is population relevant to problem
(look at other peer reviewed studyâs
justification)
⢠Distinguish the population from the sample
8. Sampling Frame and Sample
Universe:
theoretical
population to
generalize to
Population: largest
target population
from universe you
have access to or
âSampling Frameâ
Original Sample
Final Sample
Attrition
9. Survey
Item
Old (oneâs own SE) Change in item New (perceptions
of othersâ SE)
1
I feel that I have a
number of good qualities.
âI haveâ to âshe hasâ I feel that she has a
number of good
qualities.
2
I feel I do not have much
to be proud of.
âI do notâ to âshe
does notâ
I feel she does not
have much to be
proud of.
3
On the whole, I am
satisfied with myself.
âI amâ to âshe isâ and
âmyselfâ to âherselfâ
On the whole, she is
satisfied with herself.
âŚ
If you need to amend instrument, use a change
matrix; do not create your own instrument!
10. Materials: Informed Consent
⢠State purpose of project
⢠State procedure and how long it will take
⢠State voluntary nature of participation
⢠State risks (if any)
⢠Have them sign or state that by filing our
survey they are agreeing to participate.
12. Constructs vs. Variables
Constructs are the invisible abstract things weâre measuring (e.g.,
personality), while variables are the way weâre assessing
(measure/operationalize) those invisible things.
Constructs and variables need to be precise (is personality
measured by introversion scale or by conscientiousness scale?)
e.g., Intelligence is a construct, while the number of words
remembered is a way of assessing intelligence.
e.g., Personality is a construct, while the scores on an
introversion/extroversion test is a way to assess an aspect of
personality.
13. Variables
Operationalize Constructs, Make Distinction between
IVâs/DVâs, and Describe Level of Measurement
Example:
Does Empathy differ by group (alcohol abuse vs. no alcohol abuse)?
Empathy is my dependent variable and Group is my independent
variable.
Empathy (my construct) is measured by scores on the Faux Pas task.
Alcohol abuse (my construct) is measured by 5 or more drinks in one
day.
Empathy is a ratio-level variable measured with scores ranging from 0-
25, while my Group variable is a nominal-level (categorical-level) variable
because participants re in one of two groups (alcohol abuse group or
not in alcohol abuse group).
15. Validity
⢠Internal Validity: IV causes a change in the DV (not time or
other covariates, etc.
⢠External validity: can be generalized to the population
⢠Construct validity: does the scale measure the theoretical
construct
⢠Translational validity:
⢠Face validity (items are reasonable)
⢠Content validity (items match the domains of interest)
⢠Criterion related validity: measures behave as theory predicts
⢠Convergent validity: how close the variable aligns with the
construct (use EFA)
⢠Concurrent validity: construct relates to established
instruments
⢠Predictive validity: measure can predict an outcome
(GPAâIncome)
16. Reliability
Internal consistency: Cronbach alpha. Average
inter-item correlation
Inter-rater: if interval, correlate; if
dichotomous, kappa
Test-retest: administer same test at two times
Split half: divide instrument into 2 parts and
calculated totals, then correlate totals.
17. Brief Review: Units of Analyses
Quantitative (e.g., Age)
Nominal-level (Latin for name). Gender (M/F), Grouped (Old =
65+, middle age = 36-64, young = 35 or younger). Assign any
number of groups (old = 1, middle = 2, young = 3).
Ordinal-level is ranked (Latin for showing order). GPA (A-F), or
age (group 1 = age 1â15, group 2 = age 16-25, group 3 = age 25-54,
group 4 = age 65+)
Interval/Ratio-level (also named scale or continuous; Latin for
[equal] space [between numbers]). What is your age today in
years? ____ (a number from 1-105)
18. Types of Methodology Models
Theories explain phenomena,
Models represent Phenomena.
⢠SEM and Path models
⢠Regression models (linear, logistic, ordinal)
⢠ANOVA models (repeated-measures)
⢠Time-series
⢠Heirarchical Linear Models (HLM)
⢠Correlational Models
20. Data Collection Method --
Overview
⢠Describe the procedures used to administer
the materials to the participants
⢠Remember to be as detailed as necessary
so someone can literally replicate your
study
21. Data Collection
Qualitative
⢠Procedure for accessing participants
⢠Selection of data collected
⢠Number and duration of interviews
⢠How and when data is collected
⢠How data is recorded (e.g., hand
notes, audiotaped)
⢠Role of researcher-relationship to researcher
Quantitative
⢠Procedure to administer measures
22. Pilot Test (only if you develop
instrument)
⢠Detect potential issues in the instrument
⢠Allows you to get feedback and to finalize
your survey/interview items
⢠Makes sure participants understand survey
items
⢠Assess typical responses to survey items or
interview questions (were participants
comfortable, long-winded, defensive, etc.
⢠Can you access data?
23. Data Analysis Plan
⢠Quantitative: Describe the analysis plan used
to test each hypothesis, the assumptions of
the statistical analyses, and a justification of
the appropriateness of the analysis for each
research question.
⢠Qualitative: Describe how the data will be
analyzed (or thematized)
⢠Phenomenological
⢠Case Study
⢠Grounded theory
24. Sample Size
Quantitative
Varies by type of
statistical analysis
1. Research questions
in statistical language
2. Level of
measurement of
variables
3. Determine
statistical analysis
4. Conduct power
analysis
Qualitative
Varies by which
qualitative approach
is taken
⢠Phenomenological
⢠Case Study
⢠Grounded theory
25. Assumptions, Limitation, Delimitations
Assumptions: Discuss things out of your control
about the population and design, then justify
assumptions (e.g., participants will answer
honestly.)
Limitations: Are out of your control and
describe weaknesses in design, threats to
validity (e.g., generalizability).
Delimitations: Are in your control and relate to
choices you will make to narrow the scope of
the study (e.g., variables, research questions).
26. Ethical Considerations
⢠Describe informed consent procedures
⢠State whether your study will be anonymous
or confidential with respect to the participants
⢠Describe considerations for children or
vulnerable participants
28. See yourself graduate in 2014!
1-1 Personalized Dissertation Consulting
877-437-8600
Info@StatisticsSolutions.com
29. Thank you for
your participation
and attention!
Join us for our next webinar on
Wednesday, April 23rd at 8:30pm ET.
Results and Discussion
Chapters, Editing, and Defense