Mike Morgan presented a method called visual data collection for taking structured notes during user research. It involves creating visual guides with screenshots of prototypes and predefined annotation symbols to log participant actions and feedback. Data from multiple participants is aggregated into a tally sheet using visual connectors to identify patterns. The method aims to help researchers stay focused on goals and analyze data faster. Attendees watched a demo then tried annotating a prototype themselves to experience the benefits of visual structuring for note-taking.

1. Good afternoon everyone! It’s great to be here at ReCon 2018. My name is Mike Morgan. I
am a UX Researcher with Bloomberg.
Today we’re going to have some fun with straws! Just kidding.
Actually, today I am going to show you a method of data collection I have been using over
the past few years during mostly early-phase prototype research that I thought would be
interesting to share with you all.
It’s something I call visual data collection. You could also call it visual note-taking.

2. Here’s the agenda. I’ll briefly talk about the problem we’re trying to solve.
I’ll give an overview of some common note-taking tools and techniques used in the field
Then I’ll show you the method details for visual data collection.
We’ll save the best for last and do a fun data collection exercise where you’ll first watch a
play-by-play using the technique, then you’ll get to try it out yourself.
But first a little bit about myself.

3. I’ve been in working in the IT industry for about 20 years in various roles. I like to say that I
sort of backed into the field of UX research not realizing at the time that what I was doing
had a name to it.
I’ve worked in financial services for most of my career.
First as an engineer, then a business analyst, and ultimately as a UX researcher.
I also worked at ADP’s Innovation Lab for 4+ years doing UX research for their next
generation HCM software solutions.
Like many people in our field, I have a motley educational background.
I earned my Bachelor’s degree in English and Creative Writing from Binghamton University.
I also have an MBA in Finance/Strategy from NYU Stern and an M.S. in HCI from Iowa
State University.
I enjoy writing about UX research methods as well as UX strategy and innovation.
I have a column called Discovery on UX Matters which focuses on insights learned about
UX research for early-phase concept testing studies.
I’ve also written for some other publications as well.
But enough about me. Let’s talk about the problem we’re trying to solve.

4. We collect LOTS of data when we do UX research. Why shouldn’t we? It’s OUR job, right?
We hope to learn something new with each participant. Or identify a pattern of behavior that
may yield some insights, some design implications.
Much of what we collect during sessions might be useful and actionable but does it always
align with our initial research goals?
The problem is we don’t always know at the time whether or not we may need a piece of
data. So what do we do. We collect it “just in case.” Then what happens? We start to collect
more and more “just in case” data.

5. Eventually, the deluge of data that we collect may become unwieldy.

6. [person at desk]
Then when it comes time to analyze the data we may end up analyzing things beyond the
initial scope of our research goals.
[clock]
With an unlimited amount of time to do UX research having more data is not a bad problem
to have. After all, we will learn things we did not know before about our product and our
users.
The more data we can collect the more we can learn about our users, right?
But the reality is projects DO have limited amounts of time to get things done.
Deadlines need to be met so that designers can get their changes to engineers in time
before their sprints start.
Researchers need to ensure they stay focused on the original research goals or risk
[show dog]
stakeholder frustration and disengagement.

7. The note-taking approach I am going to show you today which I call visual data
collection will allow you to:
-Stay focused on your research goals
-Deliver reporting faster
-Will make your notes easily traceable in case you need to revisit them
[dog]
Ultimately you’ll end up with more satisfied stakeholders
Before I show you the method and we do a fun interactive note-taking exercise, I want to
do a brief review of note-taking tools and approaches that are currently being used in the
field.

8. According to UX researcher Whitney Quesenberry, most note-taking techniques have little
structure in place. There’s the less structured physical tools we use like post-its, notepads
and whiteboards. As you move along this line data contains a little more structure in the
guise of spreadsheets like Microsoft Excel and cloud-based solutions like Air Table. There
are solutions that utilize tagging of free-form data like Evernote or digital brainstorming tools
like Mural. And then along the far right you have power tools that log tasks like Ovo or tools
like Nvivo that can analyze large amounts of qualitative data in various formats.
To collect all of this data, we may gravitate towards using our own tools and techniques for
note-taking. I’m sure everyone has their favorite tool for taking notes.

9. There is no universal manner in which to take notes. According to UX Consultant and UX
Matters columnist Jim Ross, what note-taking approach we use really depends on
factors like the subject matter of your research, or, the research method being used,
among other things.
I think the note-taking approach we decide on is usually an after-thought or a given when
undertaking a research effort.
We already have in our minds the note-taking approach we’d like to use.
Ultimately, the way you record and organize the data should allow you and your teams to
make sense of it.

10. Techniques to make sense of your data as suggested by consultancies like Nielsen
Norman Group include topical notes-- recording observations and coding on post-its, then
affinity mapping the data to identify the main issues and themes.

11. Chronological logs is a more structured method for taking notes. It involves coding and
timestamping of events while logging observations.
Path analysis is a more visual method for taking notes. It allows us to to analyze the flow a
user takes within an experience to identify problems. We can analyze patterns across all
users in a more visual fashion with path analysis.
The Visual Data Collection method is similar to each of these-- it involves recording events
and tagging them. Similar to path analysis, visuals are an integral part of the note-taking
process.

12. So what is visual data collection? Here is a definition which sums it up. [read the slide]
In order to conserve paper I typically collect data using a digital tablet and stylus. The only
paper I end up printing out is when I am ready to do my final analysis.
I’ll show that to you as a part of this working example.

13. In a moment I’ll explain to you what these mean.

14. So what does it look when we put all of these components together?
Here is a snapshot of a visual data collection guide for a user study involving a fictional
search engine site.
In the visual data collection guide you will see a series of screenshots of the prototype that
depict the flow that a user would experience.
This is what you would call “the happy path.” Here you see the initial search screen then
the ensuing search results screen.
Along the right hand side are corresponding questions that relate to the visuals.
For easy reference I put the annotation key used for this study along the right hand side.
These are the shorthand symbols recorded during the sessions that allow for datasets to be
consistent across participants within a study.
For the sake of simplicity there are only 5 annotations listed here.
The annotations here include: the sequence of actions, elements the participant did or did
not understand as was intended by the design (or conceptual model), what the participant
selected in the prototype, participant feedback or suggestions and user expectations.
So if we were to interpret this user’s data, it appears that they first selected the text box,
entered in “Bruce Willis” then hit search. They expected after selecting the search button to
view a list of results. This participation did not understand what the “Roll the Dice” button
meant.
After being told in the session what it was, the participant suggested calling the button

15. something else.

16. So imagine you’ve logged all of your data into their respective visual data collection
interview guides. Now you have to aggregate them into one guide so you can analyze
them for patterns and themes.
Since we consistently applied annotations across the participants, analyzing the results
should be much easier.
The tally sheet is a master document which should include all of your participant’s data.
The annotations for the tally sheet are slightly different than the individual VDC guides.
Responses for each of the questions and annotations will include the participant’s number
as denoted by a “P” and their number enclosed in a circle.
This enables you to trace back any data directly to that participant.
Also, connectors are used to link together similar findings.
Connectors can include participant-to-response (orange). So in this case we see that two
participants did not understand the Roll the dice feature and one did.
There’s also participant-to-participant (blue) which suggests that participants exhibited the
same behaviors. In this example all 3 participants’ first action was to select the search box.
And lastly there’s participant-to-action sequence (purple). So in this case the first
participant’s second action was to select the search button.
With this very basic example you can see how easy it becomes to analyze data.

17. I do want to acknowledge that this method is not without its challenges.
-Websites that have many ways to accomplish the same task become difficult to design
data collection guides for. The number of pre-defined responses could become
overwhelming.
Rather than crafting questions with pre-defined responses, it might be worth considering
leaving open-ended questions then documenting the path they took.
-Another challenge is space. As you have more and more participants, space within the
tally sheet becomes a challenge.
Typically I’ve used this method successfully for up to 10 participants. Beyond that, the
increasing density of annotations becomes harder to discern.
-Lastly, subject matter without a UI may not benefit from this method. Questions which
are more exploratory in nature might require further detail, transcription and coding.

19. Now it's time to watch it in action.
Pay attention: because you’re going to be trying this out in a minute!
We're going to do this in two parts:
First, you’re going to watch the annotations in a scripted, play-by-play mode alongside the
screen share of the prototype.
Then, for the rest of the video you’re going to try it yourself!
So here’s the scenario…[read the scenario on the slide then play the video]
As an observer and note-taker you should be clued into what the “happy path” is before
seeing the scripted portion of the video.
So I’ll walk you through that now.

21. Let’s watch the video demo.

22. Now at your table are handouts of the visual data collection guides as well as the
annotation key for reference during the session.
Use the annotation key when recording your observations. Observe the participant’s
actions and their corresponding sequence.
Watch for what they do and do not understand.
Answer the pre-defined questions.

23. Now that you’ve experienced the method, I’d like you to pair off with someone else and
compare results.
Try to interpret each others annotations. Then answer these questions
Then afterwards, I’m going to ask a volunteer to share their experience with the audience.

24. Before I conclude this presentation I just want to highlight potential areas for future research
using visual data collection.
This method has proven useful for early-phase prototypes. It has served as a useful way to
collect lots of interaction data and impressions from participants for early-phase prototypes.
Could it prove useful for collecting data for summative studies like usability testing?
How about products that don’t have a UI? How would this method capture notes without
any screenshots of UIs to annotate? Things to explore for sure.
And lastly, I’ve been using digital tools like pencils and tablets to collect data saving much
in terms of using paper up.
If we all used digital methods for collecting data as opposed to paper printouts how much
could we save in terms of $$ and trees?

25. If anyone wants to discuss this method further or has any ideas about extending this
method to other applications I’d love to chat.
Here’s my info. Thanks for listening and the opportunity to present this method. Enjoy the
rest of your conference!