Loyalty program Lookup Modal User Study

ScoreCard Lookup Modal
USABILITY STUDY | 4.3.17
Page 1eCommerce User Experience
Usability Study | 4.3.17
Tim Broadwater
Table of Contents
Executive Summary............................................................................2
Test Users.............................................................................................3
Test Tasks.............................................................................................4
Test Summary......................................................................................8

Executive Summary
eCommerce Purchase & Loyalty Product Management has requested user research on the automated ScoreCard Lookup
Modal. Purchase & Loyalty Product Management are concerned that the ScoreCard Lookup Modal that display on the order
review page is stopping users from completing their purchase, and negatively affecting conversion.
PdM Research Objectives
• Do users understand what this modal means to them?
• Are there any confusion points around the language ‘Apply ScoreCard’ and ‘No Thanks!’? What do users expect to
happen if they click these links?
• Would seeing this modal stop any users from converting?
• Would seeing this modal drive customers to call into customer service and why?
• Overall do users appreciate us doing this for them or would they prefer to add their scorecard number a different way?
• Do they consider it intrusive to their shopping experience/privacy?
UX Research Methodology
Remote users were screened by analytics from the Dick’s Sporting Goods Insights Team, ecommerce search terms provided
by the SEO Team, and as being ScoreCard holders/non-customers. Users were then presented with a prototype that allowed
them to proceed through the checkout funnel, and gauge their experience and interaction with the ScoreCard Lookup Modal.

User Screening
Demographic: Desktop Computers
Age: 30–60
Income: $40K–$150K+
Web Browsers: Chrome, Safari
Are you an online shopper?
Yes : [ Accept ]
No : [ Reject ]
When purchasing items online, which device do you use
the most? Select all that apply.
Desktop : [ Must Select ]
Tablet : [ May Select ]
Phone : [ May Select ]
When purchasing items online, do you typically sign in or
use guest checkout?
Sign In : [ Reject ]
Guest Checkout : [ Accept ]
When shopping for sporting goods products, which online
retailer do you prefer? Select all that apply.
Amazon : [ May Select ]
BSN Sports : [ May Select ]
Dick’s Sporting Goods : [ Must Select / Reject ] *
Eastbay : [ May Select ]
Epic Sports : [ May Select ]
Play It Again Sports : [ May Select ]
Are you a Dick’s Sporting Goods ScoreCard Rewards
Member? (this question was only for the ScoreCard holder group)
Yes : [ Accept ]
No : [ Reject ]
User Demographics
User Online Shopping Experience
ScoreCard Holders Non-DSG Customers
• 84% Amazon
• 16% Play It Again
Sports
• 12% Eastbay
• 6% BSN Sports
• 1% Epic Sports
• 100% Amazon
• 5% Play It Again
Sports
User Age
• 52% Ages 30–39 Yrs.
• 24% Ages 40-49 Yrs.
• 24% Ages 50–59 Yrs.
• 66% Ages 30–39 Yrs.
• 17% Ages 40-49 Yrs.
• 17% Ages 50–59 Yrs.
User Sex
• 70% Female
• 30% Male
• 77% Female
• 23% Male
User Web Browser
• 88% Chrome
• 12% Safari
• 90% Chrome
• 10% Safari
(50 ScoreCard holders and 40 non-Dick’s Sporting Goods ecommerce
customers were tested) *
Test Users
The UX Team ran 90 remote users – screened into two groups * – in the United States through a ScoreCard Lookup Modal
usability study. Users were screened by the following criteria and represent the following demographics:

Tasks 1–7
1. If you haven’t already been redirected to the study, it can be accessed at http://bit.ly/21IBwZ9.
2. This page serves as a reference guide for the credit card information and billing address you can use in the checkout
process. Click on ‘Go to Shopping Cart’ link to continue.
3. You should now have two tabs open within your internet browser. Please verify that you have a tab that shows the credit
card and billing address information and a second tab that shows the shopping cart.
4. Show how you complete the checkout process using Guest Checkout. Please use the credit card and billing address
information that was provided to you in the beginning of this study.
5. Did you complete the checkout process and successfully submit your order?
• 88% Yes, I submitted my order
• 12% No, I did not submit my order
• 82% Yes, I submitted my order
• 18% No, I did not submit my order
6. On the review page, before placing your order you were asked if you’d like to apply your
ScoreCard Number. Did you apply the ScoreCard?
• 84% Yes, I applied the ScoreCard Number
• 16% No, I did not apply the ScoreCard Number
• 70% Yes, I applied the ScoreCard Number
• 30% No, I did not apply the ScoreCard Number
7. When you were on the Order Review webpage a modal popped-up. Did you understand the
purpose of this modal, and what it was for?
• 60% Yes, I understood the purpose of the modal
• 28% I’m not sure of the purpose of the modal
• 12% No, I didn’t understand the purpose of the modal
• 37% Yes, I understood the purpose of the modal
• 35% I’m not sure of the purpose of the modal
• 28% No, I didn’t understand the purpose of the modal

Tasks 8
8. Briefly describe what function the modal that popped-up on the Order Review webpage served.
Write your response below. (below are the filtered ScoreCard Holder responses for Task 7)
No, I didn’t understand the purpose of the modal
“I don't know what pop-up this question refers to - the only one I remember seeing was the pop-up that asked if I would like
to add my scorecard information to my order. I am not sure what function that pop-up performs, and I don't think any pop-
ups came up in my checkout or order review process aside from the one that said add your scorecard.”
“I guess it was to quickly check if I was okay with my order and was ready to checkout.”
“I do not understand the term modal, so I'm not sure I can fully answer this question.”
“If in reference to the score card pop up then I was aware of some kind of loyalty card but I wasn't sure if that was the same
card I was given in the other tab. If the sample credit card info was indeed a score card I wasn't sure what number I should
use or where to find the number code.”
“I do not remember a modal popping up at all.”
“Again like I said, I didn't notice a "modal" pop up on the webpage, other than one for the scorecard.”
I’m not sure of the purpose of the modal
“I remember seeing my order, and I remember seeing the shipping info. I'm not sure what the modal was.”
“The only pop-up I remember, which I thought was on the Pay page but must be mistaken, told me it had found my
ScoreCard account # and asked if I wanted to earn rewards points on this order. It gave me two options, yes and no, and I
chose Yes.”
“It basically gives you an overview of what you ordered. It allows you to verify all information. It gives you the date you will
receive it. It also is able to pull your ScoreCard information.”
“The pop up automatically found my rewards card number and asked if I wanted to apply it. I did.”
“To apply my scorecard points to my purchase.”
“Was it to enter the rewards card? I am unsure of this question, but if it was the prompt to enter the rewards card number, it's
function was to alert you that the rewards card for your account was found based on information I provided. This allowed
me to enter the rewards card to receive purchase points.”
“I do not recall a modal pop up.”
“I believe it served as a way for me to use my Dick's scorecard number.”
“To remind me to apply my reward card for points.”
“I truthfully do not remember or did not notice a modal on the order review page.”
“I'm not sure what modal is being referred to. If it was about getting points and using points from my scorecard, it came up
to let me know I could use my scorecard on this order and asked if I wanted to.”
“Letting me know that my scorecard acct had been found and asked if I wanted to apply or use my points for this
transaction.”
“On the order review page I do not recall a pop-up. There was simply my information to review my order before payment.
There was a module pop-up once i selected submit.”
“I did not understand the modal.”

Tasks 9–10
9. Please click on the following link: http://bit.ly/2nvLIrZ. This feature in the process you just
completed is an automated ScoreCard Lookup. Please rate how disruptive you found the
automated ScoreCard Lookup feature. [ 1 Not Disruptive – 5 Very Disruptive ]
• 1.28 Mean
• 1 Median
• 1.57 Mean
• 1 Median
10. Why did you assign this rating? Write your response below. (below are responses for a ratings of 3–5 for Task 9)
“Typically pop-ups at this point in the process is for applying
or creating a new account. I generally click through these
because I don't want to create accounts.”
“As directed, I considered myself to be a basic, non-registered
user. Thus, I did not see the need to apply for a ScoreCard,
since I was not under the impression I should join the site
as a member. It was easy to skip the ScoreCard application
button.”
“The score card pop up was helpful but some more detail
could've been given to know that I was using a score card,
could get, why inputting a Score Card number would be
beneficial as in I can get points towards future purchases,
and WHERE the 12 digit number on the Score Card is.”
“The fact that it was a pop up and not included in the
checkout screen threw me off. It took me a second to figure
out if it was a sign-up for the company's credit card or not.”
“It blocked certain things, description, but it did not block the
price or quantity.”
“I just felt like it wasn't needed and I was a little annoyed. I
just wanted to proceed with the checkout.”
“Because it was an unexpected pop up and not something
requested by the customer.”
“I don't like things/windows to pop up while I’m in the middle
of checking out.”
“I just feels like your getting pressured or pushed into
something. I don’t like that.”

Tasks 11–12
11. Please rate how useful you found the automated ScoreCard Lookup feature.
[ 1 Not Useful – 5 Very Useful ]
• 4.92 Mean
• 5 Median
• 4.25 Mean
• 5 Median
12. Why did you assign this rating? Write your response below. (below are responses for a ratings of 1–3 for Task 11)
“It was useful that it popped up IF I knew what it is and
WHERE the Score Card number is.”
“I don't understand the purpose of the scorecard.”
“It's helpful, but because of when it popped up and what
the title says ("Apply") I took that as a create account. Had
I looked more closely at it, I would have clicked Yes to apply
my purchase to my account.”
“I did not find it useful. I'm still not exactly sure what it is for,
but pretty sure I didn't need it.”
“I liked that it was able to locate my score card information
based on my name even though I was using guest checkout.”
“The main think I am concerned about is my order. I don’t
want anything other than my order.”
“I did not realize that was an option.”
“The reason that I didn't find the scorecard useful is that I
was not looking to take advantage of the potential discounts
available by using points that I had stored up.”
“It's nice that it automates that for you, but if you're trying to
checkout quickly and aren't interested in earning rewards it
can be annoying.”
“As I'm not certain of what the benefits are of using my
ScoreCard, I rated this in the middle. If the ScoreCard
benefits are useful to me, then I would rate this feature much
higher.”

Test Summary
After experiencing the ScoreCard lookup modal in a prototype, and being prompted to apply their ScoreCard order, 16% of
Dick’s Sporting Goods ScoreCard holders were confused about if they applied their ScoreCard to their order. This percentage
greatly increased to 30% for non-customers.
After experiencing the ScoreCard lookup modal in a prototype, and being prompted to apply their ScoreCard order, 40%
of Dick’s Sporting Goods ScoreCard holders were unsure or didn’t understand the purpose of the modal. This percentage
greatly increased to 63% for non-customers.
When asked to describe the function of the ScoreCard lookup modal, common themes from users were:
• Users are not sure of the function or purpose of the modal
• Users skipped passed it or missed the modal, dismissing it as a pop-up
• Users thought the modal was signing them up for something
When asked rate the ScoreCard lookup modal in regards to it being disruptive, high scores stated:
• Users thought that this modal was to create an account, or sign-up for a card, pressuring them to do something, and
they didn’t want to do so
• Users thought there were enough instructions or that instructions were unclear
• Users felt that it was a blocking content
• Users didn’t seem to see the need for the modal
• Users just wanted to proceed to check out, found it annoying, or would just click pass it anyways
When asked rate the ScoreCard lookup modal in regards to it being useful, low scores stated:
• Users wanted it to be next to the ScoreCard widget
• Users thought the verbiage ‘apply’ that it wanted them to create an account
• Users were wondering why it was doing this if they were checking out as a guest
• Users didn’t understand the purpose of the modal, and didn’t think it was useful
• Users stated that they weren’t concerned with the ScoreCard, they just wanted to checkout, and that they weren’t
looking for that when checking out
UX Recommendations
When exploring wireframes, prototype iterations, or additional user testing for optimization, the UX Team suggests:
• Employing content strategy for messaging
• Experimenting with different verbiage for CTAs
• Exploring in-page integration versus modals
• Designing different UI options for functionality
• Determining options for moving feature in the checkout funnel

Loyalty program Lookup Modal User Study

Recommended

Recommended

More Related Content

More from Tim Broadwater

More from Tim Broadwater (20)

Recently uploaded

Recently uploaded (8)

Loyalty program Lookup Modal User Study