Conference Abstract: Did you know that there are measurement experts who spend their lives researching the validity of different question styles and formats? How ’bout we spend 30 minutes reviewing some of the most relevant findings in survey methods research!? During this presentation, you will learn about fascinating topics such as the validity of true/false questions, when to use “none of the above” style questions, and which question styles will help you assess the information you are actually trying to asses (read= minimize bias and error)! Participants will be asked to vote on sample question strength, and recraft “before’s” into highly valid “after’s.”

1. Wendy Turell, DrPH, CHCP
Senior Director, Outcomes and Analytics
PlatformQ Health Education

2. • Measurement about how people think or act gathered via Q&A process
• Can be used in experimental, quasi-experimental, and non experimental research
• Initially developed in the 1920s, now have theories and principles that guide survey
science
• Survey Methods = commonly use in CEHP
• Surveys:
• Can be administered via self administered questionnaires or interviews
• Paper/pencil, verbally, online
• Are most often quantitative (but can be qualitative)
• Are a standardized measurement instrument
• Replicable
• Can measure:
• Attitudes and opinions
• Behaviors
• Motivations
• Emotions
• Satisfaction

3.  Measurement Validity
 How accurately the instrument measures the construct you are trying to measure
 Examples:
 A Ruler = High measurement validity
 Measurement reliability
 How consistently your instrument measures what you are trying to measure.
 Does your testing instrument generate the same result in similar circumstances?

4.  Many errors “bias” research results
 Examples: sampling error, nonresponse error, self-presentation/social desirability bias,
acquiescence bias, etc.
 Today: Focus on measurement error
 The difference between the estimate produced and the true value because respondents
gave inaccurate answers to survey questions.
 Occurs when respondents are unable / unwilling to provide accurate answers
 May be due several factors……..including poor question design
 Does the question adequately measure the idea or concept of interest?
 If not = specification error (low construct validity)

5.  Pilot your questions
 Psychometric review
 validate your questions against an external construct (e.g., correlate with a measure that
is known to measure the construct of interest very well)
 examine internal consistency of constructs
 To begin with………..write solid questions by using question structures that have
been shown to elicit more valid & reliable responses!

6. TIP 1:
 Question:
 Measurement studies have shown that 5 option MCQ formats are more reliable and
valid than dichotomous answer “true/false” style questions:
a. True
b. False
 True/false; Yes/no; Agree/disagree
 Probability of guessing correctly = .5 (dichotomous) versus .2 (5 choice MCQ)
 “Acquiescence” = tendency to agree as a response style
 When guessing, examinees tend to lean towards “true” > “false”
Grosse, Martin E.; Wright, Benjamin D. (1985) Validity and Reliability of True-False Tests. Educational and Psychological
Measurement, 45(1): 1-13
Norcini JJ, Swanson DB, Grosso LJ, Webster GD. (1985). Reliability, validity and efficiency of multiple choice question and patient
management problem item formats in assessment of clinical competence. Med Educ. 19(3):238-47.

7. TIP 2:
 Examples:
 All of the Above, None of the Above, A and E, All except D
 More likely to lead to measurement error
 More challenging to think of good distractor answers
 All of the Above:
 When studied, responses skew more towards “all of the above” response (typically artificially
elevating scores)
 Loss of discriminatory power in reviewing other response options
Cuing Effect of "All of the above" on the Reliability and Validity of Multiple-Choice Test Items P. H. Harasym,E. J. Leong, C. Violato, R.
Brant,and F. L. Lorscheider. Evaluation & the Health Professions, March 1998; vol. 21, 1: pp. 120-133.

8. TIP 3:
 Novel targets in clear cell RCC include all of the following EXCEPT:
a. PD-1
b. PD-L1
c. MET
d. WNT pathway
 Which of the following prognostic features is associated with WORST overall survival?
a. Low Ki-67 proliferative index
b. Morphology of small cells resembling follicular leukemia
c. Presence of 3 or more complex karyotype abnormalities
d. Translocation involving chromosomal 11
 “not”, “except”, “least likely” etc.
 Flip cognitive reasoning
 > Taxing = > measurement error

9. TIP 4:
 Questions that contain multiple concepts
Do you recommend changes in diet and exercise for all of your overweight patients?
In the past month, how frequently have you self-monitored your blood sugar and blood
pressure?
How satisfied are you with the accuracy and delivery of the content?
Do you desire more live and online CME activities in oncology?
Rewrite:
Do you recommend changes in diet for all of your overweight patients?
Do you recommend changes in exercise for all of your overweight patients?

10. TIP 5:
 I can rarely differentiate a diagnosis of patients with schizophrenia from those
with schizoaffective disorder.
 how to define “rarely”
 My health has improved since participating in the educational session.
 how to define “health”
 I have visited my doctor in the past 6 months.
 how to define “doctor” – do other healthcare providers count?
 how to define “visit” – do phone/email consults count? What if you just saw the nurse?

11.  Which question format do you prefer?
 Question 1: Basal insulin should provide about ______ of daily insulin requirement.
a. 10%
b. 25%
c. 50%
d. 75%
 Question 2: In the evaluation and diagnosis of a person with mild cognitive impairment, which of
the following biomarkers are considered most likely to indicate that MCI is due to AD:
a. A positive CSF Aβ biomarker only
b. A positive Aβ biomarker and a positive tau, FDG, or sMRI biomarker
c. Positive tau and FDG biomarkers
d. A positive FDG biomarker only

12.  Which question format do you prefer?
 Question 1: Some ER/LA opiates have eliminated risk of abuse.
a. True
b. False
 Question 2: Patients with an unprovoked VTE are at risk for a recurrence for
what amount of time?
a. 3 months
b. 6 months
c. 12 months
d. Indefinitely

13.  Based on what you learned in this session, rewrite this question:
 Which is NOT your favorite color crayon?
 Yellow
 Blue
 Green
 Brick Red

14.  Based on what you learned in this session, rewrite this question:
 The following outcomes level is the most difficult for me to evaluate and report:
 Level 3: Knowledge
 Level 4: Competence
 Level 5: Performance
 Level 6: Patient Health
 Level 7: Community Health

15. Thank you & good luck!
Contact: wturell@platformq.com

17. Wednesday April 8th, 2015
1:30 PM ET
5 Tips to Make You a Survey Measurement Rock Star
Did you know that there are measurement experts who spend their lives
researching the validity of different question styles and formats? How ’bout we
spend 30 minutes reviewing some of the most relevant findings in survey methods
research!? During this presentation, you will learn about fascinating topics such as
the validity of true/false questions, when to use “none of the above” style questions,
and which question styles will help you assess the information you are actually
trying to asses (read= minimize bias and error)! Participants will be asked to vote
on sample question strength, and recraft “before’s” into highly valid “after’s.”