Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Improving Your Surveys and Questionnaires with Cognitive Interviewing

198 Aufrufe

Veröffentlicht am

Do you use surveys or questionnaires in your work? If so, it is important to be sure you are collecting the data you need. Too often, we write surveys without much attention to how participants will understand and respond to our questions. Just like forms, instructions, websites, or any other product, users often have their own interpretations of our content.

We can improve our survey questions with a method called cognitive interviewing. This method was developed to improve questionnaires for large survey companies and government organizations. The method is similar to usability testing, as it evaluates how a user experiences a survey, but there are also some critical differences between the methods. In this session, we will describe cognitive interviewing, and show how it is similar to and different from usability testing. We will discuss how to conduct cognitive interviews, what data to collect, and how to analyze the results.

Presented by Jean Fox, Jennifer Edgar and Scott S. Fricker

Veröffentlicht in: Präsentationen & Vorträge
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Improving Your Surveys and Questionnaires with Cognitive Interviewing

  1. 1. 1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Improving Your Surveys and Questionnaires with Cognitive Interviewing Jean E. Fox Scott S. Fricker Jennifer Edgar U.S. Bureau of Labor Statistics UXPA 2018 June 28, 2018 All views expressed in this paper are those of the authors and do not reflect the views or policies of BLS.
  2. 2. 2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Thank you Scott Fricker Jennifer Edgar
  3. 3. 3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Agenda  Surveys in UX work  Why we need Cognitive Interviewing (CI)  Background on CI  How CI is similar to and different from Usability Testing (UT) So you can get started with CI  More ways that CI can help!
  4. 4. 4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov What is a Survey?  A series of questions given to respondents to collect information from them “Questionnaires” and “polls” have same purpose  Systematic Ask the same thing to all participants Can get data from many people easily  Self- or Interviewer-administered
  5. 5. 5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Surveys in UX Work  We can use surveys On their own With other methods – Usability Testing – Card Sorting – Field Observations
  6. 6. 6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Standard UX Surveys  Known UX surveys that have been tested Reliable Valid Sensitive  Examples System Usability Scale (SUS) (Brooke) SUPR-Q (Sauro) SUMI, WAMMI (Kirakowski) QUIS (Norman)
  7. 7. 7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov BUT…  If standard surveys can’t address our issues We write our own.
  8. 8. 8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Home-Grown Surveys  Benefits— Target the information we want Easy to write (right??)  BUT… Are you sure you are measuring what you want? How do you know?  AND You know how to design interfaces, but you still run usability tests And you still find problems.
  9. 9. 9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Cognitive Interviewing Make sure you are measuring what you want to measure
  10. 10. 10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov So, What is Cognitive Interviewing?  It’s kind of like Usability Testing for surveys. Participants answer survey questions You talk to them about their response process Identify problems with the survey  But there are some important differences.
  11. 11. 11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Background on Cognitive Interviewing  Developed in the field of Survey Methodology  Goal is to learn how respondents: Understand the question Retrieve the information Decide what answer to give Respond to the question  Problems can occur in any of these stages
  12. 12. 12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Types of Problems CI Can Uncover  Complicated Instructions  Confusing or vague question wording  Jargon  Inappropriate assumptions (e.g., the question doesn’t apply to your respondent)  Unexpectedly sensitive topics  Questions that are difficult to answer  Answers that are difficult to recall
  13. 13. 13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Just to Give You an Idea…  How many UX professionals are in your group? What is your answer? How did you determine your answer? Who did you include as a “UX Professional”? Exclude? How did you define your “group”? What does it mean to be “in” your group? What time frame did you include?
  14. 14. 14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Testing Overview – Both UT and CI  Define research goals, create materials, etc.  Recruit participants  Conduct study Informed consent Complete task Debriefing Incentives  Analyze data  Make recommendations
  15. 15. 15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Testing Goals  UT Uncover usability problems and potential solutions  CI Ensure the question/survey is capturing the intended information
  16. 16. 16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Moderators  UT & CI: Similar skills and training Work comfortably with all participants Interact in an unbiased manner React and adjust the test session as needed Analyze and interpret data
  17. 17. 17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov When to Test  UT & CI Ideally, start early and continue through development stages Goals are different at different stages – Early – Test wireframe / concepts – Midway – Test features / questions – Late – Test overall product / survey
  18. 18. 18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Participants  UT and CI Target specific populations, determined by research goals Typically use convenience samples  UT Often 5-10 (per user group) Or more for a quantitative study  CI Typically 20-30
  19. 19. 19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Moderating Sessions  UT In-person or remote Moderated or unmoderated  CI In-person or over the phone Almost always moderated
  20. 20. 20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Tasks  Type of Tasks UT: Complete tasks using the product/website CI: Answer survey questions  Scenarios/vignettes to provide context UT: You are a school counselor, and you want to know the education requirements for accountants. CI: Pretend this is your high school transcript, and answer these questions about your courses.
  21. 21. 21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Thinking Aloud  UT and CI Information about participant’s processes, thoughts, and reactions Pros and cons, decide based on research goals
  22. 22. 22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Debriefing Questions / Probes  To learn about participants’ thoughts and reactions Helpful for UT Essence of CI
  23. 23. 23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Debriefing Questions / Probes  Can be done During the task – Get thoughts as participants are working – But this can interrupt their work Post-Task / Post-question(s) – Get feedback on specific tasks/questions Post-Test / Post-survey – Get overall impressions
  24. 24. 24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Goals of Questions / Probes During the Task  UT and CI Explore unexpected behaviors, responses, and reactions Understand why participants are struggling
  25. 25. 25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Sample Questions / Probes During the Task  UT and CI I noticed you hesitated, can you tell me what you are you thinking? What are you looking for?
  26. 26. 26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Goals of Post-Task Debriefing Questions / Probes  UT Why participants did what they did during an individual task How they felt about their experience Scripted or spontaneous  CI What participants were thinking as they answered a specific question or group of questions How they decided what to do Scripted or spontaneous
  27. 27. 27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Sample Post-Task Debriefing Questions / Probes  UT How easy or difficult was the task? How confident are you that you completed the task?  CI In your own words, what was the question asking? What does <term or phrase> mean to you? How did you decide on your response?
  28. 28. 28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Goals of Post-Test Debriefing Questions / Probes  UT and CI Overall impressions about the product / survey What participants liked and didn’t like Any recommendations they have Scripted or spontaneous questions
  29. 29. 29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Data Collected Data Usability Testing Cognitive Interviewing Qualitative • Observations • Comments • Debriefing responses Quantitative • Task times • Success rates • Satisfaction ratings • Etc • Actual responses to survey questions, in context of the debriefing information
  30. 30. 30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Analysis for Both UT and CI  Qualitative data – look for patterns and themes across participants  Quantitative data UT: look at means by user group, use statistics as appropriate CI: review responses in context of the answers to the probes
  31. 31. 31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Outcome of Analysis  UT Identify usability problems – Effectiveness • Tasks that are difficult to complete • Features that are hard to use – Efficiency • Activities that took too long to complete – Satisfaction Make recommendations – Ways to fix the usability problems
  32. 32. 32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Outcome of Analysis  CI Identify problems with – Understanding the questions – Retrieving the information – Deciding what answer to give – Responding to the question Make recommendations – Rewording of questions – Rework skip patterns – Provide context for interpreting survey results.
  33. 33. 33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Getting Started with CI  Whenever you are asking users questions, consider testing the questions first. For usability testing, use Cognitive Interviewing to evaluate your questions during the pilot test. For other surveys, consider starting with a review by your UX colleagues or actual respondents You want to know – Do they interpret the questions the way you want? – Can they answer the questions? – Do the response options make sense?
  34. 34. 34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov CI Can Do Even More for UX work!  Evaluate web content, instructions, or other text products Can users understand it? Does it mean what you think it means?  Evaluate other types of forms Do people know what goes in each field?
  35. 35. 35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Conclusions  It’s important to be sure that our surveys are asking what we want them to ask.  CI can help you do that  What you know about UT already may make it easy for you to incorporate CI into your research  To learn more about CI, see: Willis, G. B. (2004). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage Publications.
  36. 36. Contact Information 36 — U.S. BUREAU OF LABOR STATISTICS • bls.gov Jean E. Fox Bureau of Labor Statistics Fox.Jean@bls.gov
  37. 37. 37 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Usability Testing Cognitive Interviewing What are the objectives of the methods? Identify usability problems and potential solutions Understand how respondents interpret survey questions and the challenges they face completing a survey When in development would you use the method? Anytime in development Who conducts the tests? Typically User Experience professionals, possibly with a formal coursework related to usability testing Typically Survey Methodologists, often with informal training in cognitive interviewing Who participates in the tests? Participants are from the target population(s) Smaller sample sizes for qualitative tests, larger for quantitative tests Generally aim for 20-30 participants
  38. 38. 38 — U.S. BUREAU OF LABOR STATISTICS • bls.gov @jeanharrisfox Usability Testing Cognitive Interviewing What methods are used? Participants complete tasks (answer survey questions) while their experience is observed in some way Participants complete tasks assigned to them, often with scenarios providing context for the task Participants complete surveys with their own responses In-person or remote testing Usually in-person testing Sometimes moderated Always moderated Can be done in lab or at respondent’s location May use think aloud What data are collected? Quantitative and/or qualitative data Generally qualitative data only Focus is on performance and user satisfaction Focus is on cognitive processes while completing the survey

×