A baseline usability study with design recommendations

1. Last.fm Baseline Usability Study
CBS Interactive Research/Analytics/User Experience
September 5, 2008

2. Overview of contents
Why we conducted a Usability test
Where we tested – Study Facility
How we tested – Methodology
Who we tested – Participant Profiles
What we saw – Observations
What should we do – Recommendations
Next Steps
2

3. Why we conducted a Usability test:
• To establish a usability and user experience baseline of Last.fm
• To establish the foundation for a method of observation and evaluation
• To discover strengths and weaknesses of the current website
• To provide recommendations for design alternatives
3

4. Overview of contents
Why we conducted a Usability test
Where we tested – Study Facility
How we tested – Methodology
Who we tested – Participant Profiles
What we saw – Observations
What should we do – Recommendations
Next Steps
4

5. Where we tested - Study Facility
• CBS Television City, Las Vegas – NV
• Usability tests were conducted in one of the adjacent focus group rooms
similar to the one pictured above.
5

6. Overview of contents
Why we conducted a Usability test
Where we tested – Study Facility
How we tested – Methodology
Who we tested – Participant Profiles
What we saw – Observations
What should we do – Recommendations
Next Steps
6

7. How we tested - Methodology
• CBS Television City staff recruited 10 participants.
• A screener was used to recruit: young, tech savvy, online music
enthusiasts.
• Each session lasted about one hour.
• Participants were given an opportunity to freely explore last.fm
• Afterwards, a moderator guided each participant through a series of Tasks:
 Play music or video (interact with controls)
 View photos
 Register for the site
 Find similar artists
 Add music to library
 Interpret site content
 Compare registered user’s “home page” to “profile page”
7

8. How we tested – Methodology, continued…
• Participants started their session one of two ways:
1. From an Artist page (to simulate entry from a Search engine)
2. From the Last.fm home page (to simulate entry from an advertisement or word-of-mouth)
• The start task was alternated between sessions to avoid potential ordering
affects.
• An Eye Tracker was used to record eye fixations (areas of interest on a page)
during the initial exposure to the Last.fm Home page.
• Participants were asked to talk aloud while they interacted with the site.
8

9. Overview of contents
Why we conducted a Usability test
Where we tested – Study Facility
How we tested – Methodology
Who we tested – Participant Profiles
What we saw – Observations
What should we do – Recommendations
Next Steps
9

10. Participant Profiles
• Gender mix: 6 Men, 4 Women
• Ages: ranged from 18-30 years of age (average =23)
• Education:
– 6 participants had some college education (4 were currently students)
– 3 participants had earned four year college degrees
– 1 participant had completed high school (no collegiate course work)
• Marital Status: 9 single, 1 married
• Music spending: participants reported spending an average of $66.00 on
digital music over the last six months - though this amount varied widely
among participants (ranging from $10 to as much as $200).
• Occupations: Retail Sales Representative, Student, Construction Worker,
Director of Membership & Marketing, Information Systems, Counselor
10

11. Participants – Young, tech savvy, online music enthusiasts
Jonathan - CA Paul - CA Amanda - IL Svetlana - PA David - VA
Age: 20 Age: 21 Age: 19 Age: 22 Age: 30
Fav Band: Linkin Park Fav Band: Three Days Grace Fav Band: Jacks Mannequin Fav Band: Less Than Jake Fav Band: Kenney Chesney
Recent Visits: Yahoo! Music, Recent Visits: Yahoo! Music, Recent Visits: iTunes, Recent Visits: AOL Music, Recent Visits: Yahoo! Music,
iTunes, MySpace, Facebook Pandora, Facebook Facebook, MTV Yahoo! Music, iTunes, iTunes, MySpace
Download.com, MySpace,
Facebook, MTV
Sergio - CA Megan - FL Andrew - TX Chris - TX Kelly - PA
Age: 18 Age: 30 Age: 21 Age: 27 Age: 21
Fav Band: DragonForce Fav Band: Dave Matthews Fav Band: Circa Survive Fav Band: Sparta Fav Band: 50 Cent
Recent Visits: AOL Music, Recent Visits: iTunes, Napster Recent Visits: AOL Music, Recent Visits: iTunes, Recent Visits: Yahoo! Music,
Yahoo! Music, iTunes, iTunes, MySpace, Facebook, MySpace, Facebook, MTV iTunes, MySpace, Facebook
Rhapsody, MySpace, MTV
Facebook, MTV
11

12. Overview of contents
Why we conducted a Usability test
Where we tested – Study Facility
How we tested – Methodology
Who we tested – Participant Profiles
What we saw – Observations
What should we do – Recommendations
Next Steps
12

13. Observations: Main Takeaways
• Lots of positive feedback about Last.fm:
– Comprehensive nature of the site (information and music)
– Ability to view a band’s touring information (Events)
– Accurate and contextual music recommendations
• But, even after an hour of exposure, participants did not fully or
confidently grasp the purpose and utility of Last.fm
• At the end of the test, the following comments were common:
– “I don’t know how well it works or what it does”
– “I haven’t had a chance to see what last.fm does”
• Call-to-action (especially on the Home page) did not strongly direct
participants to engage the site (do things) on their own.
• Value proposition for site membership is unknown.
• Music licensing (preview vs. full tracks) is not clear to everyone.
• Community aspects are somewhat transparent.
13

14. Observations: General
• Home and Artist page designs seemed boring to some participants - not as
flashy and eye catching as other sites.
• Too much “white space” on the site – content pushed down the page.
• Page layout/design did not seem to direct participants to take action.
• Font size for text and labels seemed to be small for some participants –
more difficult to read.
• Top navigation links are not always recognized as “navigation” and tend to
be ignored especially once a participant scrolls down the page.
• Participants were ambivalent to the brand name – Last.fm. To some it
meant “satellite radio” or a “physical AM/FM radio station”, while others
made no connection at all.
Click on the video icon to view highlights of the points made on each slide in this presentation.
(5 minutes) 14

15. Observations: Home Page
• Participants not inclined to take
action or engage the site.
• Call-to-action was weakest on the
Home page.
• The “Start Your Last.fm” copy did
not sufficiently explain what a user
is supposed to do in this field.
• A heat map of the Home page
illustrates that people see the field.
The issue is that they do not
understand how to start the
engagement.
The red areas indicate longer
eye fixations on (potential
interest) while the green areas
indicate shorter eye fixation
durations. Cleary participants
saw the field… they just didn’t
engage it.
(45 seconds) 15

16. Observations: Artist Page
• Participants seeking music were very positive of content on the site – “music
Wikipedia, credible, and comprehensive”.
• Tour/Event information was a crowd pleaser!
• Several participants compared Last.fm favorably in comparison to MySpace
and Yahoo! Music sites.
• Participants tended to read text but the site lacks stickiness (engagement) and
the call-to-action is weak.
• Player not always noticed or engaged prior to moderator prodding.
• Tags for music were noticed by some participants but were not always
perceived as accurate. Negative impression for Last.fm
• “Ban this Track” was not a useful feature and has a potentially negative user
experience if clicked on by accident – there’s no way to undo!
• While playing a track, clicking on the Radio Station link on the Player spawns a
new window and plays a second song. Two songs trip over each other!
• The concept of a radio station is not understood and often confused with real,
physical AM/FM/SAT radio stations.
(10.5 minutes) 16

17. Observations: Music Licensing
• In general, participants did not understand why some songs played in full
while others were only previews.
• Last.fm’s music catalog and licensing limitations are not appropriately
conveyed – may result in users assuming they must sign-up in order to
hear a song even though the full song is not available.
(1.5 minutes) 17

18. Observations: Registration
• The sign-up (Registration) process was easy!
• Some participants mistakenly entered both verification words in one box
(it requires two).
• Purpose and benefit of membership is unknown. Most think they:
– Will get email alerts for events or other notices
– Will be able to listen to full tracks
– Will be able to create a playlist
– Communicate with other users on the site
• Audio verification is not usable! Participants said it was:
– “Creepy”
– “Really odd”
– “If someone saw this they might exit the site completely… sounds like gibberish”.
• All participants used their personal email addresses during the study (even
though they were given an option to use a test email). Good!
(4.5 minutes) 18

19. Observations: Music Recommendations
• Participants responded positively to music recommendations.
• Recommendations were regarded as accurate and contextual by
participants.
• Last.fm won over participants with the design of showing why a band is
being recommended (bands in your library promote other bands).
(1 minute) 19

20. Observations: Community
• Even though most of the participants were active MySpace and Facebook
users, the Community aspects of Last.fm were not apparent.
• Comments on Artists by other users not important to most participants.
20

21. Overview of contents
Why we conducted a Usability test
Where we tested – Study Facility
How we tested – Methodology
Who we tested – Participant Profiles
What we saw – Observations
What should we do – Recommendations
Next Steps
21

22. Recommendations – Design Intent
• Provide clearer (more directive) call-to-action on Home, Artist, and User Home
pages for:
– Listening to music
– Discovering music, artists, and bands
– Adding music to a library at initial sign-up Mission Statement is clear and concise.
– Joining the Community Option to take a tour.
– Any other engagement on the site
Mission statement is clear and
concise. Iconic call to action
for: Listen, Get Paid, Socialize.
Prominent Sign-up and short
explanation for registration.
Mission Statement is clear and concise –
prominent text for: music, videos, and photos.
22

23. Recommendations – Page Design
• Create design alternatives for Home and
Artist pages to test ideas and discover if a
design promotes intended user action.
– A/B Testing
– Usability Testing
• Consider more engaging art work and
visuals that are “vibrant and fun”.
• Remove the white space at the top of the
Home page to move content higher up on
the page.
– The current text had little or no value to
participants and simply takes up space.
• Labels for member areas are dull and
white/on/gray – easily ignored by the eye.
• Display the header navigation links in a
more prominent manner (they are
displayed in a subdued form currently)
• Consider using a larger font size or
provide a variable text size feature.
• Link the Last.fm logo to the Home page
not a user’s Home page.
23

24. Recommendations - Player
• Consider ways to visually promote the Player via color choice and/or
placement on the page – the Player was often overlooked, maybe due to
color contrast.
• Strongly consider providing a Pause button. Participants overwhelmingly
preferred Pause over Stop!
• Reconsider the value of “Ban this track”.
– Participants did not consider it useful and banning a track has a potentially negative side-
effect if clicked on by mistake.
– If retained, provide an “Undo” ban capability.
• The Player design should not intentionally play two songs at the same time.
– In the case where a user is listening to a track and then clicks on the Similar Artist Radio
Station, stop the first track and load the radio station in current page – do not spawn a
new browser window.
Results in two songs playing at the same
24

25. Recommendations – Player, continued…
• The Radio Station icon resembles a RSS or news aggregator icon which
confused some participants. It also looked like a podcast to some.
Address in design.
• Consider ways to explain why some songs are previews while others are
full tracks.
– Idea: color code tracks that are previews vs. full tracks to orient user.
• Make Last.fm’s music licensing agreements and limitations readily
apparent to users.
25

26. Recommendations - Registration
• Define and orient the user to the value of a
Membership on Last.fm Opportunity to reinforce value
and purpose of Membership.
• Remove the current audio verification feature
on sign-up.
– It was not usable and adds a negative impression of
Last.fm (challenges trust and confidence with the site)
– Revisit if a tested and viable audio option becomes
available.
• Separate the visual word verification boxes on
the Sign-up page. Some participants tended to
put both words in one box – an error.
• More directive text for Verification feature.
• Labels and copy for directions are hard to read
(light gray on white).
• Although not examined in this study, consider
how to position the difference between a
Membership and a Subscription on Last.fm.
26

27. Recommendations, Misc.
• Explain that Tags are generated by the
Community.
– Some participants thought the tag clouds
were inaccurate based on names/words
present which could skew a user’s
impression of Last.fm.
• Allow for better alignment of Similar
Artist labels with their corresponding
picture thumbnail.
• Consider combining a user’s Home page
and Profile page.
– Some participants (especially users not
familiar with MySpace or Facebook did not
understand the difference).
27

28. Recommendations, Misc.
• Periodically check Search functionality:
– The Search utility failed to find very popular bands (i.e., Chris Brown and Usher) during at least
one session- a potentially huge negative impression for Last.fm
• Be consistent with the default display
for Events and Year.
– When “Events” was selected from the left hand
navigation for the band “The Fray”, the year
2005 was the default. Intuitively the user
would expect the most recent events (2008) to
be the default.
28

29. Recommendations, Misc.
• Page treatment was inconsistent. Keep labels consistent for pictures. Currently the
terms Images and Photos are used interchangeably. Be consistent.
Artist page
Image page
29

30. Overview of contents
Why we conducted a Usability test
Where we tested – Study Facility
How we tested – Methodology
Who we tested – Participant Profiles
What we saw – Observations
What should we do – Recommendations
Next Steps
30

31. Next Steps
• Review study findings with the last.fm design team.
• Consider new design alternatives and strategies.
• Schedule iterative usability testing of design alternatives – Oct 2008?
• Expand the testing audience to include a wider user demographic.
• Complete competitive landscape study after brand positioning focus
groups.
31