The document summarizes two localization studies conducted by Jennifer Fabrizi and Anna Bradley. The first study in 2009 tested a financial services website with Spanish-speaking and Chinese-speaking participants. Key findings included differences in how the groups valued translation quality, content, and visual design. A second 2010 study tested design changes made in response to the first study. This remote and unmoderated study found differing feedback from the Spanish and Chinese groups compared to the first study, likely due to cultural factors and the absence of a moderator. Overall, the studies demonstrated that in-language usability testing is possible and provided lessons about considering cultural influences.

1. Localization Research:
If we can do it, so can you!
Jennifer Fabrizi
Anna Bradley
Boston UPA Conference
May 25, 2011

2. Introduction
Jennifer Fabrizi
Lead of User Experience Research and
Design, Massachusetts Mutual Life
Insurance Company
Anna Bradley
Currently Interaction Designer at
Memento Security; previously at
Massachusetts Mutual Life Insurance
Company

3. Agenda
• Our story
• Background
• Study One: 2009
• Study Two: 2010
• Why the findings were different
• What we learned overall

4. This is the story about how 2 user experience practitioners
figured out how to conduct usability tests with
in-language participants and moderators
and learned a lot in the process
Our story
Jen Fabrizi Anna Bradley

5. Background

6. http://geoffallan.blogspot.com/
New Multicultural Markets

7. http://online.wsj.com/article/SB10001424052748704604704576220603247344790.html#project%3DCENSUS_SHIFT_2011%26articleTabs%3Darticle

8. Marketing Research showed…
Hispanic Americans
• Have a lack of trust in financial institutions
• Depend on family and friends for advice
Relationship development with agent is key

9. Marketing Research showed…
Chinese Americans
• Are averse to debt accumulation & personal
gratification
• Planning for longer term: protecting future
generations
• Rely on experts for advice
Agent as expert, company stability and financial
performance are key

10. Why in-language studies?

11. Study One: 2009

12. Recruiting
Multicultural Testing = nuance of localization
• 5 Hispanic-Americans with Spanish as first language
• 5 Chinese-Americans with Mandarin Chinese as first language
• Pertinent pre-screener requirements:
• Emigrated to U.S.A. prior to age 16
• Current age 35-65
• Vendor support:
• Bentley Design and Usability Center for review
• New American Dimensions in Los Angeles to recruit and hire
moderators and translators

13. Conducting the Study
Testing Room:
Spanish or Chinese speaking
moderator and participant
Observation Room:
Court-certified synchronized interpreter,
note takers, and observers
Morae
recorder
Morae
recorder
Synchronized
Translator
Moderator
Participant
One of us
taking notes
Hosts and observers

14. Conducting the Study
Testing Room:
Spanish or Chinese speaking
moderator and participant
Observation Room:
Court-certified synchronized translator,
note takers, and observers
Morae
recorder
Morae
recorder
Synchronized
Translator
Moderator
Participant
One of us
taking notes
Hosts and observers
Lots of Talking Points
on this slide!
Multicultural users = nuance of localization?
Details of the study:
• In-language moderator; usability script translated into Spanish and
Mandarin
• Used a court-certified synchronized interpreter to provide live,
real-time translations into English into the control room
• Traveled to Los Angeles; used our mobile Usability Lab (Morae on 2
laptops) to record the participant sessions and translations
• Used Camtasia to remove the audio of the in-language sessions
and replace it with the translation audio
• Used Morae Manager to pull clips to play for stakeholders; also
provided full in-language .wmv files
• One glitch: Morae wouldn’t connect with a 3rd laptop running
Observer via the vendor’s network – learned that our mobile lab
had to have network ports opened!

15. What we learned
Chinese Hispanic
decision-
making
premium, coverage, customer service, reputation, branch locations/convenience
expectancy expected link to lead to fully-translated site
translation
quality
excellent with few exceptions (i.e., mutuality)
awareness low
general useful and informative
findability of
link
high low
translated
content
appreciated translated content as
data for decision making
partial translations seemed to be less
appreciated; translations seemed to serve
emotional need vs. functional role
visual design looks professional; Asian faces
evoked positive connection; liked
clean colors and information-focus
liked detailed information; colors were
cold

16. What we learned
Chinese Hispanic
decision-
making
premium, coverage, customer service, reputation, branch locations/convenience
expectancy expected link to lead to fully-translated site
translation
quality
excellent with few exceptions (i.e., mutuality)
awareness low
general useful and informative
findability of
link
high low
translated
content
appreciated translated content as
data for decision making
partial translations seemed to be much
more negative; translations seemed to
serve emotional need vs. functional role
visual design looks professional; Asian faces
evoked positive connection; liked
clean colors and information-focus
liked detailed information; colors were
cold
Warning!!
Our study was designed to test for users’ behavior when trying to find
links and figure out what they meant.
Participants also gave us feedback about attitudes toward aspects of
the site, including quality of translation, translated content, use of
color, and overall visual design. These are all important aspects of a
localization strategy for a site. We didn’t have such a strategy, so we
probed about these attitudes and presented these findings as
directional only to our internal partners.
If the original study requirements had been to do a behavioral and
valid attitudinal study, we would have used different recruiting
methods (i.e., a much larger sample size!!) in combination with a
different study design (probably remote-unmoderated via UserZoom),
followed by qualitative usability.
Our advice is to always make sure to report your findings in the correct
context!

17. Recommendations

18. Recommendations
A Typical Usability Error?
Setting Users’ Expectations
The in-language links didn’t set users’ expectations well in terms of
what the site provided. In-language content was not a full-translated
site; it was simply high-level content to reach out to these specific
markets in their own languages.
The links said “Spanish” and “Chinese”.
Recommendation 1 = change text to “More about MassMutual” in
their own languages
Recommendation 2 = change text of the title of this section from
“Browse by Feature” to “In-language Information”
Through a second parallel mass affluent study not covered in this
presentation, we cleared out the “junk drawer” by finding locations in
the information architecture for everything else.
Because findings regarding other aspects of localized design were
directional only, we could not recommend changes to visual design
treatment, content voice (which we heard was good feedback on),
how much of the site to translate, translated menus, menu design, etc.
Our recommendations aimed at being reasonable, cautious,
and inexpensive.

19. Study Two: 2009

20. Design Changes Made
• As a result of the 2009 study the link text was changed
to be more in line with the true experience
• The text was changed to say “More about MassMutual”
in Spanish, Mandarin Chinese and Korean
• In addition other links were moved to the menus (based
on a separate usability study with mass affluent
participants)

21. Follow up study of site changes

22. Setup and Recruiting
• The recruiting criteria were the same for this study as
for the 2009 study though the recruiting was done via a
UserZoom™ recommended recruiter
• This study was un-moderated,
done through UserZoom™
• Prototype had one working link
and one follow-up page
• Remote and un-moderated for
broad geographic access
• Questions after the clicks.

23. In-language Considerations
• All text presented to the user was translated into their
language
• UserZoom has a feature which translates the default
text and instructions
• All questions were asked in-language
• The Spanish-speaking participants replied in Spanish
• The Chinese-speaking participants replied in English.
We assume this was due to a lack of a Chinese
keyboard.
• Responses were then translated by the same team
who translated the original text

24. Findings
• All Spanish participants and several Chinese participants noted
that the in-language links should be located in the upper right
hand of screen.
• The Chinese participants found it easier to locate the in-language
link due to Chinese characters standing out more amongst the
text.
• All Spanish participants were fine with what they found when
they clicked on the link. They said it met their expectations.
• The Chinese participants were split. One participant mentioned
wanting to see the same homepage translated. One thought
they saw more fluff than content. Others did not comment.

25. Why were the findings different?
• The un-moderated study had nearly reversed results
from the 2009 study
• In an environment without another person present the
Spanish speakers stated they were fine with the site.
The Chinese participants on the other hand felt free to
state that they did not like the lack of a full translation.
• We believe that the un-moderated nature of the study
changed how both groups presented their opinion
• The altered text of the link also helped set the
expectation for the Spanish speakers, but it did not
help the Chinese speakers

26. What we learned
• In-language studies can be done on a reasonable
budget.
• There are tools out there that make it easy to splice the
synchronized translation in.
• Cultural differences have to be taken into account but
are influenced by adaptation to American culture.
• We were surprised by the different results but after
understanding them in a cultural context they make
sense.
• If we can do it, so can you!

27. Questions?

28. Thanks for coming! Contact us!
Jennifer Fabrizi
Lead of User Experience Research and
Design, Massachusetts Mutual Life
Insurance Company
Anna Bradley
Currently Interaction Designer at
Memento Security; previously at
Massachusetts Mutual Life Insurance
Company
jfabrizi@massmutual.com
www.linkedin.com/in/jenniferfabrizi
twitter.com/@jenniferfabrizi
abradley@mementosecurity.com
http://www.linkedin.com/pub/anna-
margulis-bradley/3/b1a/7b3