Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Advanced Card Sorting and Mixed Method IA Validation

689 Aufrufe

Veröffentlicht am

This presentation grew out of my experience working with an information architecture that was incomplete and lacked generative research to understand the users. I was tasked with finding ways to validate the IA while providing useful user insight for future design changes. To meet the goal, I studied card sorts in depth and developed useful techniques for problems that often arise but are seldom discussed.

I also explored the use of other IA validation techniques such as task-based tree tests and click tests to complement and cross-validate card sort findings. Through the use of mixed methodology within-subject study design, card sort data can be more easily understood and validated by the outcomes of tree tests and click tests.

This presentation will provide practitioners with information about card sorting in advanced situations. Specifically it will cover card sorting when applied to complex hierarchies, multiple audiences, and information architecture validation.
Presented by Caron Garstka at UXPA 2018

Veröffentlicht in: Präsentationen & Vorträge
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Advanced Card Sorting and Mixed Method IA Validation

  1. 1. www.nicecompany.comCOMPANY NAME Caron Garstka UXPA 2018 Card Sorting and Mixed Method IA Validation
  2. 2. www.nicecompany.comCOMPANY NAME A little about me Information Architecture (IA): A quick review IA validation methods Mixed method IA validation Case Study 01. 02. 03. 04. 05. C ON TEN TS Conclusion
  3. 3. www.nicecompany.comCOMPANY NAME A little about me 01
  4. 4. www.nicecompany.comCOMPANY NAME A l i ttl e a bo u t m e … Who I am… • Caron Garstka • Senior UX Researcher at Express Scripts • Masters of Science in Information from University of Texas at Austin What I do… • Conduct research with patients who have chronic, complex, or life-threatening conditions • Help shape their digital experiences using their feedback My goal is to make patient lives easier
  5. 5. www.nicecompany.comCOMPANY NAME Information Architecture (IA): A quick review 02
  6. 6. • It is the creation of a structure for a website, intranet, application, or other project • Allows users to understand where they are, and where the desired information is in relation to their position • Results in the creation of site maps, hierarchies, categorizations, navigation, and metadata W h a t i s In fo r m a ti o n Ar c h i te c tu r e ( IA) ?
  7. 7. Information architecture is about helping people understand their surroundings and find what they’re looking for, in the real world as well as online. – Information Architecture Institute W h a t i s In fo r m a ti o n Ar c h i te c tu r e ( IA) ?
  8. 8. A Ph a se d Appr o a c h to IA D e ve l o pm e n t There are multiple steps to creating and maintaining an IA Research Strategy Design Implementation Administration Card Sorting lives here!
  9. 9. A Ba l a n c e d IA Re se a r c h Appr o a c h Documents Data types Content objects Metadata Volume Existing structure Business goals Funding Politics Culture Technology Resources Constraints Audience Tasks Needs Information seeking behavior Experience Vocabularies C o n t e n t C o n t e x t U s e r s
  10. 10. How is it categorized? How is it displayed? Where should it be stored? What information is relevant at a given time? Is the information necessary? What are they looking for? What’s important to them? What are they familiar with? C o n t e n t C o n t e x t U s e r s A Ba l a n c e d IA Re se a r c h Appr o a c h
  11. 11. A Ba l a n c e d IA Re se a r c h Appr o a c h Heuristic evaluation Metadata and content analysis Content mapping Benchmarking Background research Presentations and meetings Stakeholder interviews Technology assessments Search log and clickstream analysis Use cases and personas Contextual inquiry User interviews and user testing C o n t e n t C o n t e x t U s e r s
  12. 12. IA Context UsersContent A Ba l a n c e d IA Re se a r c h Appr o a c h
  13. 13. www.nicecompany.comCOMPANY NAME IA Validation Methods 03
  14. 14. Content audits Establishing starting point Surveys Gathering opinions Field studies, diaries, contextual inquiry, interviews, focus groups Learning about users Web traffic statistics, clickstreams Online behavior Be fo r e do i n g IA va l i da ti o n … Do the following methods before starting validation research.
  15. 15. IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names
  16. 16. IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into groups and give groups a name; typically done prior to an architecture being put in place • Allows the researcher to document a user’s mental model and inform the IA
  17. 17. IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into defined categories • Allows the researcher to document a user’s mental model and understand how well the proposed IA matches their model
  18. 18. IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into researcher-defined categories but have the option to create their own • Allows the researcher to document a user’s mental model and allows users to participate in building the IA
  19. 19. IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users work one after another to complete card sort • Users modify as needed and pass modifications to next user • Allows the researcher to rapidly arrive at an organizational model
  20. 20. IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users navigate a hierarchy using only top-level link names. • Allows the researcher to see how findable items are within the hierarchy
  21. 21. I A V a l i d a t i o n R e s e a rc h Me t h o d s Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users locate items within a static design • Allows the researcher to see if the top level IA is easily findable within the context of the design
  22. 22. C ompari son of IA Research Methods Method Research Type Activity Learn About Project Stage Open card sorting Generative Organizing Mental models Organization Labeling Relationships between items Early user research Closed card sorting Evaluative Organizing Mental models Organization Labeling Information architecture design Hybrid card sorting Generative Evaluative Organizing Mental models Organization Labeling Relationships between items Early user research Delphi card sorting Generative Organizing Mental models Organization Labeling Relationships between items Early user research Reverse card sorting (Tree testing) Evaluative Finding Organization Labeling Findability Information architecture design Click testing (or usability testing with low- fidelity prototypes) Evaluative Finding Organization Labeling Findability Navigation design Interface design Interface design
  23. 23. Pr a c ti c a l T r i a n gu l a ti o n [Triangulation is an] attempt to map out, or explain more fully, the richness and complexity of human behavior by studying it from more than one standpoint. - Cohen and Manion (1986)
  24. 24. www.nicecompany.comCOMPANY NAME Case Study: Accredo.com 04
  25. 25. www.nicecompany.comCOMPANY NAME Accredo.com Accredo.com 1. Stakeholders wanted to validate a proposed IA on the Accredo.com website 2. Conduct a closed card sort to catch any glaring issues The Perceived Need
  26. 26. www.nicecompany.comCOMPANY NAME The Real Need Expanded scope… 1. How well the proposed IA matches the user’s mental model 2. How findable items are within the proposed IA 3. How UI helps or obscures IA 4. Areas for improvement 5. Validate the proposed patient IA 6. Learn how potential users group content 7. Evaluate the findability of topics within the website structure 8. Evaluate the homepage design and whether potential users can fine the expected entry point for specific information or functionality within the design And there wasn’t much time… Accredo.com Accredo.com
  27. 27. www.nicecompany.comCOMPANY NAME • Site caters to multiple audiences • Users are distributed • Site IA is different pre and post login Focus on one side of the experience first. Conduct study remotely, asynchronously, and without moderation. Allowed for fast data capture without having to moderate each session. Accredo.com Accredo.com Other Challenges
  28. 28. Appr o a c h Tree Testing Closed Card Sorting Click Testing Hybrid Card SortingOpen Card Sorting Delphi Card Sorting One method alone would not get at what the team needed.
  29. 29. Appr o a c h Used UserZoom to conduct the study. Tree Test N= 43 Median Completion Time: 25 min Click Test N= 44 Median Completion Time: 20 min Open/Closed Card Sort N= 87
  30. 30. Appr o a c h • Get at users’ mental model • Note alternative categories, item placement, and category names • Pin point hard to place items Step 01 Open Card Sort • See how users group items within given categories • See how well groupings match the proposed IA • Note alternative categories, item placement, and category names • Pin point hard to place items Step 02 Closed Card Sort • Look at where users drill into the tree • Note findability of key items Tree Test Step 03A • Look at where users drill into the design • Note expectations of finding items in design Click Test Step 03B
  31. 31. A p p ro a c h Tree Test N= 43 Median Completion Time: 25 min Click Test N= 44 Median Completion Time: 20 min Open/Closed Card Sort N= 87
  32. 32. C ard Sort Setup 1. Selected content that could be grouped 2. Selected content that was at the same level 3. Selected content that was representative 4. Selected content that did not lead the activity 5. Selected content that participants would understand Maintained consistency across the open and closed card sorts.
  33. 33. Ope n C a r d So r t • Presented participants with a list of Accredo proposed IA items and asked them to sort those items into groups • Participants sorted items into groups that made sense to them • Participants created their own names for the categories the created • Some labels were modified for the card sort to add clarity UserZoom interface for open card sort
  34. 34. Ope n C a r d So r t F i n di n gs • Dendrogram: A tree diagram of hierarchical clustering to learn how users group items. • Dendorgram scale: 0 to 1; the lowest level of clusters with the highest agreement. • Euclidian distance the average distance between items. Interpreting dendrogram clusters is an art. Data Output: Dendrogram Overview
  35. 35. Ope n C a r d So r t F i n di n gs Interpretations Cluster 2: Had a runt: “Communication Information for digital support team” Cluster 1: Fairly cohesive, aligned with the hypothesized IA, and participants came up with similar names Cluster 3: High agreement for items related to prescriptions, not matching hypothesized IA, but logically made sense
  36. 36. Ope n C a r d So r t F i n di n gs Cluster 4: Cohesive group related to a Welcome or introductory section Cluster 5: Mixed assortment of forms and administration items. These items had strong agreement but did not match IA. Interpretations
  37. 37. Ope n C a r d So r t F i n di n gs Finding: Participants grouped items according to Disease FAQ, Prescriptions and Refills, and Account functions Participants were less certain on how to handle “Communication information for digital support team”
  38. 38. Ope n C a r d So r t F i n di n gs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights
  39. 39. Ope n C a r d So r t F i n di n gs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights Finding: “Communication information for digital support team” and “Register” were both highly ranked for difficulty to sort and participants thought the items could belong in multiple categories. Perhaps they would be easier to sort if given a structure. Sounds like a job for the Closed Card Sort!
  40. 40. C l osed C ard Sort • Participants were first presented with a list of Accredo IA items and were asked to sort those items into pre-named groups UserZoom interface for closed card sort
  41. 41. C l osed C ard Sort Wanted to see how well participants’ sorts aligned with proposed closed groups. Home Getting Started Conditions Find Medication Contact Us Manage your account Create an Accredo account information Symptoms Search to view Accredo drugs Communication information for digital support team Download Accredo mobile app Authorization to use and disclose health information PDF Disorder overview Phone numbers for disorder- specific assistance Order status Authorization to use and disclose health information PDF Find disease-specific support Billing call information Register Patient consent agreement and acknowledgement document Specific disease/disorder care team Phone number to refill medicine How to begin with Accredo Welcome to Accredo information Financial resources for a disorder Fax number for new prescription Refill an Rx Filling Rx for the first time Medicines for a particular disease Secure email form New patient refill process Payment mail-in address Patient forms What is a specialty pharmacy? video Refill procedure for existing patients
  42. 42. C l osed C ard Sort Fi ndi ngs Methodology • For this analysis I chose to use the average linkage dendrogram for analysis. • I set the agreement minimum to 50% - this is the minimum percentage of agreement the selected items that participants included into the same category.
  43. 43. C l osed C ard Sort Fi ndi ngs Finding: Participants seemed to have an easier time grouping “Communication information for digital support team” – this was the runt in the open sort – grouping it in with other “contact” related items. Interestingly, “Medicines for a particular disease” and “Search to view Accredo drugs” hung together, where as in the open card sort they did not.
  44. 44. C l osed C ard Sort Fi ndi ngs Items Home Getting Started Conditions Find a Medication Contact Us Highest % Register #% #% #% #% #% #% Refill an Rx #% #% #% #% #% #% Order status #% #% #% #% #% #% How to begin with Accredo #% #% #% #% #% #% Welcome to Accredo information #% #% #% #% #% #% New patient refill process #% #% #% #% #% #% Refill procedure for existing patients #% #% #% #% #% #% Manage your account #% #% #% #% #% #% Disorder overview #% #% #% #% #% #% Symptoms #% #% #% #% #% #% Medicines for a particular disease #% #% #% #% #% #% Financial resources for a disorder #% #% #% #% #% #% Find disease-specific support #% #% #% #% #% #% Specific disease/disorder care team #% #% #% #% #% #% Search to view Accredo drugs #% #% #% #% #% #% What is a specialty pharmacy? video #% #% #% #% #% #% Authorization to use and disclose health information PDF #% #% #% #% #% #% Patient consent agreement and acknowledgement document #% #% #% #% #% #% Phone number to refill medicine #% #% #% #% #% #% Fax number for new prescription #% #% #% #% #% #% Patient forms #% #% #% #% #% #% Filling Rx for the first time #% #% #% #% #% #% Create an Accredo account information #% #% #% #% #% #% Phone numbers for disorder-specific assistance #% #% #% #% #% #% Secure email form #% #% #% #% #% #% Payment mail-in address #% #% #% #% #% #% Billing call information #% #% #% #% #% #% Communication information for digital support team #% #% #% #% #% #% Download Accredo mobile app #% #% #% #% #% #%
  45. 45. C l osed C ard Sort Fi ndi ngs Items Home Getting Started Conditions Find a Medication Contact Us Highest % Register #% #% #% #% #% #% Refill an Rx #% #% #% #% #% #% Order status #% #% #% #% #% #% How to begin with Accredo #% #% #% #% #% #% Welcome to Accredo information #% #% #% #% #% #% New patient refill process #% #% #% #% #% #% Refill procedure for existing patients #% #% #% #% #% #% Manage your account #% #% #% #% #% #% Disorder overview #% #% #% #% #% #% Symptoms #% #% #% #% #% #% Medicines for a particular disease #% #% #% #% #% #% Financial resources for a disorder #% #% #% #% #% #% Find disease-specific support #% #% #% #% #% #% Specific disease/disorder care team #% #% #% #% #% #% Search to view Accredo drugs #% #% #% #% #% #% What is a specialty pharmacy? video #% #% #% #% #% #% Authorization to use and disclose health information PDF #% #% #% #% #% #% Patient consent agreement and acknowledgement document #% #% #% #% #% #% Phone number to refill medicine #% #% #% #% #% #% Fax number for new prescription #% #% #% #% #% #% Patient forms #% #% #% #% #% #% Filling Rx for the first time #% #% #% #% #% #% Create an Accredo account information #% #% #% #% #% #% Phone numbers for disorder-specific assistance #% #% #% #% #% #% Secure email form #% #% #% #% #% #% Payment mail-in address #% #% #% #% #% #% Billing call information #% #% #% #% #% #% Communication information for digital support team #% #% #% #% #% #% Download Accredo mobile app #% #% #% #% #% #% Finding: “Communication information for digital support team” and “Register” were both highly ranked for difficulty sorting and belonging in multiple categories in the open card sort. However, once given a structure, participants had higher agreement for placement. “Patient forms” has a split grouping for where participants think it belongs.
  46. 46. C l osed C ard Sort Fi ndi ngs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights
  47. 47. C l osed C ard Sort Fi ndi ngs Difficult to Sort Patient consent agreement and acknowledgement document What is a specialty pharmacy? video Authorization to use and disclose health information PDF Patient forms Refill an Rx Communication information for digital support team Download Accredo mobile app Billing call information Fax number for new prescription New patient refill process Payment mail-in address Search to view Accredo drugs Create an Accredo account information Phone number to refill medicine Refill procedure for existing patients Financial resources for a disorder Find disease-specific support How to begin with Accredo Order status Disorder overview Manage your account Medicines for a particular disease Register Secure email form Welcome to Accredo information Filling Rx for the first time Phone numbers for disorder-specific assistance Specific disease/disorder care team Top “Difficult to Sort” Items Difficult to Sort Finding: “Communication information for digital support team” and “Register” were both highly ranked for participants thinking they were difficult to sort in the open card sort. Both dropped overall in the closed card sort. But now Refilling Rx is one of the top 5 most difficult to sort.
  48. 48. Items Participants felt belonged in Multiple Categories Communication information for digital support team Order status Patient consent agreement and acknowledgement document Create an Accredo account information Phone number to refill medicine Authorization to use and disclose health information PDF Billing call information Fax number for new prescription New patient refill process Patient forms Find disease-specific support Manage your account Refill procedure for existing patients What is a specialty pharmacy? video Financial resources for a disorder Payment mail-in address Refill an Rx Register Search to view Accredo drugs Specific disease/disorder care team Welcome to Accredo information Filling Rx for the first time Medicines for a particular disease Phone numbers for disorder-specific assistance Secure email form Disorder overview Download Accredo mobile app How to begin with Accredo Symptoms Top “Multiple Categories” Items Multiple Categories Finding: Even though Refill Rx was top five most difficult to sort, it was not as highly ranked for belonging in multiple categories. Order status has also jumped up in the list. This would be a prime candidate to look at in the tree test as well to see where people start looking for it. C l osed C ard Sort Fi ndi ngs
  49. 49. C l osed C ard Sort Fi ndi ngs Home Getting Started Conditions Find Medication Contact Us Hypothesis Correct Hypothesis Correct Hypothesis Correct Hypothesis Correct Hypothesis Correct Manage your account Green Create an Accredo account information Green Symptoms Green Search to view Accredo drugs Green Communication information for digital support team Green Download Accredo mobile app Yellow Authorization to use and disclose health information PDF Yellow Disorder overview Green Phone numbers for disorder-specific assistance Green Order status Red Authorization to use and disclose health information PDF Yellow Find disease-specific support Green Billing call information Green Register Red Patient consent agreement and acknowledgement document Yellow Specific disease/disorder care team Yellow Phone number to refill medicine Yellow How to begin with Accredo Red Welcome to Accredo information Yellow Financial resources for a disorder Red Fax number for new prescription Yellow Refill an Rx Red Filling Rx for the first time Yellow Medicines for a particular disease Red Secure email form Yellow New patient refill process Red Payment mail-in address Yellow Patient forms Red What is a specialty pharmacy? video Red Refill procedure for existing patients Red
  50. 50. C l osed C ard Sort Fi ndi ngs Home Getting Started Conditions Find Medication Contact Us Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Manage your account Green Create an Accredo account information Green Symptoms Green Search to view Accredo drugs Green Communication information for digital support team Green Download Accredo mobile app Yellow Authorization to use and disclose health information PDF Yellow Disorder overview Green Phone numbers for disorder-specific assistance Green Order status Red Authorization to use and disclose health information PDF Yellow Find disease-specific support Green Billing call information Green Register Red Patient consent agreement and acknowledgement document Yellow Specific disease/disorder care team Yellow Phone number to refill medicine Yellow How to begin with Accredo Red Welcome to Accredo information Yellow Financial resources for a disorder Red Fax number for new prescription Yellow Refill an Rx Red Filling Rx for the first time Yellow Medicines for a particular disease Red Secure email form Yellow New patient refill process Red Payment mail-in address Yellow Patient forms Red What is a specialty pharmacy? video Red Refill procedure for existing patients Red Since the hypothesis works for some items but not others, we need to know how easily users can find these items given the current hierarchy. Card sorting is exploratory, tree testing is confirmatory. Time to look at the tree test…
  51. 51. T r e e T e s t Se tu p The setup for the tree test activity needed to align with the card sort and then be carried over into the click test. 1. Select items that may be difficult to sort, items you want more data on, or items we had additional questions about 2. Select easy and hard items based on location in the tree 3. Create tasks for those items
  52. 52. • Participants were presented with the top-level navigation • Participants then were asked to perform tasks using the tree • When a participant believed they found the desired information, they would select it and continue. If they were unsure where to find the information, they could select “I don’t know”. T r e e T e s t
  53. 53. Clicks Task Home Getting Started Conditions Find Medication Contact Us Overall Success Success Path Time N Sign up for Accredo Specialty Pharmacy #% #% #% #% #% #% Home > Register #%; Create an Account > Register Now #%; Filling for the First Time > Call patient services to get started #%; Home > How to get started #%; Welcome > Register Now #% 0:00 # Reorder a drug your doctor prescribed you #% #% #% #% #% #% Home > Refill an Rx #%; Contact Us > Call patient services – Refills and Orders #%; Refilling Prescriptions > Phone number to refill prescription #% 0:00 # Check where your drug is in the fulfillment and delivery process #% #% #% #% #% #% Contact Us > Call patient services – Refills and Orders #%; Home > Order Status #% 0:00 # Find information about the process for getting set up with Accredo Specialty Pharmacy #% #% #% #% #% #% Home > How to get started #%; Create an Account > Create an Accredo account information #%; Welcome > Welcome to Accredo! Information #% 0:00 # Learn about the purpose of a specialty pharmacy #% #% #% #% #% #% Welcome > “What is a specialty pharmacy?” video #%; Welcome > What Accredo provides the patient #% 0:00 # Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% #% #% #% #% #% Filling for the First Time > How to fill your prescription with Accredo for the first time #%; Home > How to get started #%; Filling for the First Time > Call patient services to get started #% 0:00 # Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% #% #% #% #% #% Refilling Prescriptions > Existing Patient – How to refill your prescription #%; Contact Us > Call patient services – Refills and Orders #%; Home > Refill an Rx #% 0:00 # Find general information about Bladder Cancer #% #% #% #% #% #% Bladder Cancer > Overview #% 0:00 # Find out the signs of having Epilepsy #% #% #% #% #% #% Epilepsy > Symptoms #% 0:00 # Learn about drug treatments associated with Hemophilia B #% #% #% #% #% #% Hemophilia B > Medications #%; Find a Medication > Search Medications Offered #% 0:00 # Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% #% #% #% #% #% Growth Hormone Replacement > Financing Your Care #% 0:00 # Learn more about Leukemia assistance programs and advocacy organizations #% #% #% #% #% #% Leukemia > Find Support #% 0:00 # Learn more about Hepatitis C's Accredo care team #% #% #% #% #% #% Hepatitis C > Accredo Team #% 0:00 # Find a drug by it's manufacturer's name #% #% #% #% #% #% Find a Medication > Search Medications Offered #% 0:00 # Find number to call to ask about a balance on your account #% #% #% #% #% #% Contact Us > Call patient services – Billing and Payments #% 0:00 # Find the mailing address to send in money to pay off an account balance #% #% #% #% #% #% Contact Us > Mail your payments address #% 0:00 # Find phone number for a specialized condition-assistance team #% #% #% #% #% #% Contact Us > Call patient services – Condition-specific assistance #% 0:00 # Reach out to the Digital Support Team to get help with the website #% #% #% #% #% #% Contact Us > Call patient services – Digital Support Team #% 0:00 # Find the information for sending Accredo Specialty Pharmacy a fax #% #% #% #% #% #% Contact Us > Send us a fax – General Patient Fax #% 0:00 # Send Accredo Specialty Pharmacy an email from the website #% #% #% #% #% #% Contact Us > Send us a secure online message – Email us #% 0:00 # T r e e T e s t F i n di n gs
  54. 54. T r e e T e s t F i n di n gs Tasks Overall Success Success First Attempt Success Multiple Attempts Non Success I don't know Percent Time Percent Time Percent Time Percent Time Percent Time Sign up for Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reorder a drug your doctor prescribed you #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Check where your drug is in the fulfillment and delivery process #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find information about the process for getting set up with Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about the purpose of a specialty pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find general information about Bladder Cancer #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find out the signs of having Epilepsy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about drug treatments associated with Hemophilia B #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Leukemia assistance programs and advocacy organizations #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Hepatitis C's Accredo care team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find a drug by it's manufacturer's name #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find number to call to ask about a balance on your account #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the mailing address to send in money to pay off an account balance #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find phone number for a specialized condition-assistance team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reach out to the Digital Support Team to get help with the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the information for sending Accredo Specialty Pharmacy a fax #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Send Accredo Specialty Pharmacy an email from the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
  55. 55. T r e e T e s t F i n di n gs Tasks Overall Success Success First Attempt Success Multiple Attempts Non Success I don't know Percent Time Percent Time Percent Time Percent Time Percent Time Sign up for Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reorder a drug your doctor prescribed you #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Check where your drug is in the fulfillment and delivery process #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find information about the process for getting set up with Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about the purpose of a specialty pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find general information about Bladder Cancer #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find out the signs of having Epilepsy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about drug treatments associated with Hemophilia B #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Leukemia assistance programs and advocacy organizations #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Hepatitis C's Accredo care team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find a drug by it's manufacturer's name #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find number to call to ask about a balance on your account #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the mailing address to send in money to pay off an account balance #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find phone number for a specialized condition-assistance team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reach out to the Digital Support Team to get help with the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the information for sending Accredo Specialty Pharmacy a fax #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Send Accredo Specialty Pharmacy an email from the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Finding: “Check where your drug is in the fulfillment and delivery process” (Order Status) placement in the card sorts did not match our hypothesis. Participants also looked in a variety of categories trying to find this item in the tree test which made for longer search times. Where would they look on a homepage image? Time for the click test.
  56. 56. C l i ck Test Setup The setup for the click test activity needed to align with the tree test so the same tasks were used. 1. Select items that may be difficult to sort, items you want more data on, or items we had additional questions about. 2. Selected easy and hard items based on location in the tree. 3. Create tasks for those items.
  57. 57. C l i ck Test • Participants were presented with static image of home screen • Participants then were asked to perform tasks and click where they think they would find information
  58. 58. C l i ck Test Fi ndi ngs “Check where your drug is in the fulfillment and delivery process” (Order Status) placement did not match IA hypothesis. Participants also looked in a variety of categories trying to find this item in the tree test which made for longer search times. Where would they look on a homepage image?
  59. 59. www.nicecompany.comCOMPANY NAME Conclusion 05
  60. 60. An a l ysi s Appr o a c h Step 01 Open Card Sort Tree Test Step 03A Click Test Step 03BStep 02 Closed Card Sort To recap how we approached analysis…
  61. 61. An a l ysi s Appr o a c h Step 01 Open Card Sort Tree Test Step 03A Click Test Step 03B • Examine the groups • Analyze card placement • Analyze labels • Analyze organizational schemes • Analyze how accurately participants have grouped content • Analyze participant comments • Analyze a closed card sort • Look at items classified as difficult to sort or belonging in multiple categories Step 02 Closed Card Sort To recap how we approached analysis…
  62. 62. An a l ysi s Appr o a c h Step 01 Open Card Sort Click Test Step 03B • Analyze success rate • Analyze directness of success • Analyze time on task • Look at card sort items classified as difficult to sort or belonging in multiple categories – how do participants approach looking for those items? • Look at runts or categories with two items – are they more easily found in the tree? Where are users looking to find them? Step 02 Closed Card Sort Tree Test Step 03A To recap how we approached analysis…
  63. 63. An a l ysi s Appr o a c h Step 01 Open Card Sort Click Test Step 03BStep 02 Closed Card Sort Tree Test Step 03A • Compare where users drill into the tree v. where they click into the UI • Look at initial entry points for the tree and first click data for the click test – are there similarities? • Note if success changes from the tree test to the click test To recap how we approached analysis…
  64. 64. T i ps fo r M i x e d M e th o d IA Stu dy Se tu p • Select items and tasks that are crucial or controversial • Ordering of method matters • Be consistent across the study • Participant fatigue is a thing –scope your work!
  65. 65. T i ps fo r M i x e d M e th o d IA An a l y s i s 1. Determine if new categories should be created or consolidated using the mental model put forth in open card sort 2. Look to confirm or deny hypotheses about categories using the closed card sort 3. See what issues of findability there are in the tree test 4. Note where people would place items in the UI in the click study
  66. 66. • Allows for data comparisons – useful in understanding convergence and divergence of findings • Collects rich, comprehensive data – provides a more complete story than a single method would alone • Combining two or more of these methods worth the cost. • Time consuming to create • Participants can easily grow tired • Data might be messy • Can be difficult to successfully integrate data Some challenges with mixed method IA validation… Adva n ta ge sC hal l enges Some advantages of mixed method IA validation…
  67. 67. C oncl usi on By spending the time constructing a more robust study design, you can create a holistic view of how the user is thinking and behaving toward your IA.
  68. 68. www.nicecompany.comCOMPANY NAME GET IN T OU C H cegarstka@express-scripts.com caron.garstka@gmail.com E-mail Thanks!
  69. 69. www.nicecompany.comCOMPANY NAME Questions!

×