3. What is this workshop about?
A research and design methodology that centers the user
4. What is this workshop about?
A research and design methodology that centers the user
How to make better decisions with better data
5. What is this workshop about?
A research and design methodology that centers the user
How to make better decisionswith better data
How to build a website that generates trust and satisfaction
25. [t iFollowKANYE WEST O
.. @kanyewest
if you put a link to my site people will be able to chop
& screw the video. this is a step towards the
possibilities of internet based video
10:46 AM - 22 Jul 2013
2,734 RETWEETS 1,604 FAVORITES
26. KANYE WEST O
@kanyewest
[ Follow
please don't diminish the user experience byjust
posting it as a normal video ...
my site
10:46 AM - 22 Jul 2013
please link them to
3,278 RETWEETS 2,165 FAVORITES
40. What is A/ BTesting?
A quantitative behavioral UX research method
for asking and answering questions about
products and services
41. A = Original Version
B = Variation 1
n =Additional Variations
42. A = Original Version
B = Variation 1
n =Additional Variations
Isolate a design variable on the page, then
serve different variations randomly to a
portion of users.
47. Who else tests their live site with live users?
Etsy
Google
48. Who else tests their live site with live users?
Etsy
Google
Twitter
49. Who else tests their live site with live users?
Etsy
Google
Twitter
and many others
50.
51.
52. Tons of active A/B tests and rampups.
www.e ty.com/llhng/99045805/ttddlt· alltn-h;at'ref• r
e o e 1:ll(dd·tAlltl'I Hat brbl• IOf> x
J: >
-User 5158774
rey Vour Aoooml Help SignOu1
I I
Etsy
"
roddrt Cort
Reddlt Allen Hat
26
I M l l
53. Matt Cutts O
@mattcutts
[W'Follow!
A/B testing can be really helpful. A classic blog post
from 2 0 0 9 : goo.gl/r45Xj Lesson: buttons get clicked
more.
4:45 PM - 15 Apr 2012
40 RETWEETS 63 FAVORITES
54. Experiments @ Twitter
Thursday,September 12, 2013 I ByAlex Roetter (@aroetter),VP, Engineering [19:14 UTC]
As we've talked about before, we're constantly working to make
it even easier to follow what you care about, connect with
people and discover something new on Twitter.
- -
1·
Tweet
To that end, we are constantly evolving the product. Some changes are visible -
they may help you protect your Twitter account or make it easier to share photos;
others are under-the-hood changes that help us suggest relevant content in real time
and make Twitter more engaging.
A common thread across recent releases has been experimentation. We've tested
various features with small groups of our 200 million users before determining what
we'll release. These tests are essential to delivering the best possible user
experience.
It's rare for a day to go by when we're not
releasing at least one experiment.
Tweet
We also experiment with features that may never be released to everyone who uses
Twitter. Those experiments are perhaps even more valuable because they help us
decide what not to do - which is important as we work to keep Twitter simple while
58. Key Quotes
Etsy
---------
"We love experiments."
Google
--------------
"A/ Btesting can be really helpful."
Twitter
''It's rare for a day to go by when we're not
releasing at least one experiment."
62. GROUP I
-rnoo[!]
@) [§]
co[ID @)
[Q]
GROUP I l
- OJ(ID @)
GROUP m [fil CIDm
- [ii 00[Q]
GROUP N
OJ[ID [ID
(ID[!]
moooo
(Q1
(Q)[)
[ID{ID
rn
(ID [fil @
63. R.L. Deininger, "Human Factors Engineering
Studies of the Design and Use of Pushbutton
Telephone Sets," Bell Systems Technical
Journal (July 1960): 995-1012.
http://archive.org/details/bstj39-4-995
73. The A/ B Process
Ask a question about your website
Research the question with user interviews
74. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
75. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
Define & run the experiment
76. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
Define & run the experiment
Collect data & analyze results
77. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
Define & run the experiment
Collect data & analyze results
Report to web committee & make decision
88. Ask a question about your website
Why are Interact click actions so low?
89. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
90. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
91. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
Connect.
92. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
Connect. Learn.
93. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
Connect. Learn. Help.
94. Ask a question about your website
Why are Interact click actions so low?
Do users understand Interact content?
Which other words could describe that button?
Connect. Learn. Help. Services.
96. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
97. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HIdidn't know that 'About' was under Interact."
98. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HIdidn't know that 'About' was under Interact."
''What am I interacting with?"
99. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HIdidn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
100. Researchthe questionwith user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HI didn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
"Connect is better than Interact, but neither are very good."
101. Research the question with user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HIdidn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
"Connect is better than Interact, but neither are very good."
Hearn doesn't work."
102. Researchthe questionwith user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HI didn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
"Connect is better than Interact, but neither are very good."
Hearn doesn't work."
"Learn doesn't really work. Ijust think, what am I learning?
103. Researchthe questionwith user interviews
Does Interact accurately describe the content that you find underneath?
HNot so much."
HI didn't know that 'About' was under Interact."
''What am I interacting with?"
"Connect is too vague."
"Connect is better than Interact, but neither are very good."
Hearn doesn't work."
"Learn doesn't really work. Ijust think, what am I learning?
I think of reading abook or something."
105. Research the question with user interviews
''Services is more accurate. Help is stronger."
106. Researchthe questionwith user interviews
''Services is more accurate. Help is stronger."
"I am not an English speaker, so I look for strong words.
I look for help, so Help is the best, then Services too."
107. Researchthe questionwith user interviews
''Services is more accurate. Help is stronger."
"I am not an English speaker, so I look for strong words.
I look for help, so Help is the best, then Services too."
"Help makes sense. When I'm in the library, and I think I need
help, it would at least get me to click there to find out what sort
of help there is."
111. Define & run the experiment
Success metrics:
click-through rate, drop-off rate, homepage return rate
112. Define & runthe experiment
Success metrics:
click-through rate, drop-off rate, homepage return rate
Google Analytics ''Experiments" for the mechanics of A/ B
113. Define & runthe experiment
Success metrics:
click-through rate, drop-off rate, homepage return rate
GoogleAnalytics ''Experiments" for the mechanics of A/ B
CrazyEgg for click data & vizualization
129. • MONTANA I L IBRARY As k th e Library 91!,
,...A!.I..._ STATE UNIVERSITY
CatSearch for articles,journ als, books and more
Advanced Search
Hours News Chat
FIND REQUEST SERVICES
Find research
materials, including
articles, books,
databases, journals ,
and course reserves
Request resources
and services,
including group
study rooms,
laptops, documents,
and books
Meet w ith us for
research assistance,
writing help, and
tech support
Today
7 am - 9 pm
Tomorrow
7 am - 9 pm
More Hours
133. The A/ B Process
Ask a question about your website
Research the question with user interviews
Formulate a hypothesis
Define and run the experiment
Collect data & analyze results
Report to web committee & make decision
138. Ask a question about your website
What are the primary actions on this page?
139. Ask a question about your website
What are the primary actions on this page?
Are users misled by the "Special Collections" link?
140. Ask a question about your website
What are the primary actions on this page?
Are users misled by the "Special Collections" link?
Is our landing page wording not concise?
141. Ask a question about your website
What are the primary actions on this page?
Are users misled by the "Special Collections" link?
Is our landing page wording not concise?
What if we streamlined our language?
146. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
147. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
"Maybe I can go to Special Collections to find things that I want?"
148. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
"Maybe I can go to Special Collections to find things that I want?"
"Honestly I don't read the page. My eyes just go past all that text."
149. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
"Maybe I can go to Special Collections to find things that I want?"
"Honestly I don't read the page. My eyes just go past all that text."
HI like search."
150. Researchthe questionwith user interviews
''What's a digital collection?"
"These descriptions don't make sense to me."
"Maybe I can go to Special Collections to find things that I want?"
"Honestly I don't read the page. My eyes just go past all that text."
HI like search."
New version: "It's clearer. There's just...less"
153. Define & runthe experiment
Google Analytics ''Experiments" for the mechanics of A/ B
CrazyEgg for click data & vizualization
154. Define & runthe experiment
Success metrics:
total searches, pages per visit, average visit duration
Google Analytics ''Experiments" for the mechanics of A/ B
CrazyEgg for click data & vizualization
155. Digital Homepage - Special Collections II
v Stopped I Filters applied I View settings
Conversions I Page Metrics ....
Aug 7, 2013 -0 ct 10, 2013
• c• cD
• c
D
•a
c • c o
v
0
All Visits
100.00%
I Explorer
Conversions Site Usage Goal Set 1 Goal Set 2 Goal Set 3 Ecommerce
Experiment stopped before winner was found
Conversion Rate .... vs. Select a metric Day Week Month 1,395 Experiment Visits
65 days of data
100% visitors included
• Original
40%
• Variation
Status: 1
Not enough data to identify
winner
20%
Au... Aug 22 Sep 5 Sep 19 Oct 3
Primary Dimension: Variation
Variation Experiment Visits ..j, Pages I Visit Avg. Visit Duration % New Visits Bounce Rate
• Original 702 6.76 00:03:37 6.88% 40.97%
• Variation 693 11.38 00:04:30 11.14% 29.66%
169. UX Lessons fromA/ BTesting
Be open to surprises.
User behavior insights can be unexpected.
170. UX Lessons fromA/ BTesting
Be open to surprises.
User behavior insights can be unexpected.
We don't have all the answers.
171. UX Lessons fromA/ BTesting
Be open to surprises.
User behavior insights can be unexpected.
We don't have all the answers.
Experimentation helps inform decisions.
175. Amanda L. Goodman
@godaisies
[ W Follow ]
Just sent out my first A/B testing email newsletter campaign.
Let's see what happens...
4:18 PM - 31 Jul 2014
t.• * 1
177. "At Darien Library, we are using A/B testing to discover how to
design email newsletters that get opened and acted upon. Our
most recent test of changing a subject line generated 10%
additional opens.Then by adding a digital service to an email,we
increased usage of the service by 15 1%"
178. GroupA· Winner! Group B
Mistakes were made. You Are What You Read!
abuse complaints O
total reclp ents 942
successful dellveries
bounces
91S(97 1 )
27 ( 9 )
recipients who opened
average times ,opened
total times opened 18
last n date
recipients who clicked
unique URUredpient dicks 22
s.
22
305(33.3Y:)
1 4 um
*
0 abuse compl
nts
942 total redp ems
(97 7'6}920 successful deliveries
(2 ) 22 bounces
(17 3?.>) , 59
recipients who opened
average times opened
total tlmes op ned
' 3
213
/27115 1:16PM last open date
(2 ) 20 recipients Who clicked
28 un que URUrecipient chdcs
i., clicks/unique o n
31 total clicks
1n1ns 1:1sPM
(0.0 *) 0
last click da te
times forwarded
0 forwarded
opens
unsubscribes
0
*
*
*1127/t S8 1SPM
clicks/unlque open
total dicks
****
16(2. )
last click date
times forwarded O(0
121611.!S'.!8?11.1
forwarded opens O
unsubscribes 0
Winner And Open Rate Champ on
Sent 1215114 S:OOPM
ce TryGroup B(loser)
179. - t
- Bohyun Kim
@bohyunkim
[ Follow ]
@godaisies is now talking about doing an A/B testing using
MailChimp to improve the reach of their library newsletter.
#alamw15 #uxig
10:07 AM - 1 Feb 2015
t.• *2
183. Other A/ BTesting Applications
Digital Signage
Physical Signage
MobileApp Notifications
184. Other A/ BTesting Applications
Digital Signage
Physical Signage
MobileApp Notifications
The Library building itself
185. Open Hardware &Ubraries
Imagine having a Google- Analytics- style dashboard for your Library
bui ding: number of visits, what patrons browsed, what parts of the Library
were busy during which parts of the day, and more. Measure the Fut ure is
going to make that happen by using simp le and inexpensive sensors that can
col ect data about bui ding usage that is now invisib le. Making these
invisib le occurrences explicit wi l al ow Librarians to make strategic decisions
that create more efficient and effective experiences for their patrons.
189. BuildingAnalytics
Which stacks get the most traffic
Which stacks have high traffic but low circ rates
What is the most popular piece of furniture in the library
190. BuildingAnalytics
Which stacks get the most traffic
Which stacks have high traffic but low circ rates
What is the most popular piece of furniture in the library
How many people look at a book display
191. BuildingAnalytics
Which stacks get the most traffic
Which stacks have high traffic but low circ rates
What is the most popular piece of furniture in the library
How many people look at a book display
"Once a library gather s data, it can conduct A/ B tests by creating
different arrangements and measuring the impact. I haven't been
this excited by a library project in quite some time."
- Aaron Schmidt
203. - Jeremy G Snell
@jgsnell
[" # Follow
.@benbizzle "Websites have about a 4 year life cycle."
#il2013
12:37 PM - 28 Oct 2013
2 RETWEETS 2 FAVORITES
• t.+ *
204. [" # Follow- Jeremy G Snell
@jgsnell
.@benbizzle "Websites have about a 4 year life cycle."
#il2013
12:37 PM - 28 Oct 2013
2 RETWEETS 2 FAVORITES
• t.+ *
FollowBen Bizzle
@benbizzle
@jaclark @pcsweeney @jgsne ll @hello_scott_ I was
suggestion 4 years for total redesign. Small
enhancements during that time are a must.
2:52 PM - 28 Oct 2013
1 FAVORITE
• t.+ *
221. For Librarians experienced in UX
A/ Btesting is a valuable method that
can integrate into existing UX programs
222. - Followlindsey mclean
@lmclean001
Better user experience means I spend less time
teaching students how to use a website, eresource,
catalog, etc. #il2013 #internetlibrarian
3:18 PM - 29 Oct 2013
5 RETWEETS 3 FAVORITES
• t+ *