Our talk at CHI2015 in Seoul, South Korea. Find more information at www.kotarohara.com . YouTube: https://youtu.be/isqsYLkX9gA Makeability Lab: http://www.cs.umd.edu/~jonf/ Microsoft Research: http://research.microsoft.com/ ABSTRACT Language barrier is the primary challenge for effective cross-lingual conversations. Spoken language translation (SLT) is perceived as a cost-effective alternative to less affordable human interpreters, but little research has been done on how people interact with such technology. Using a prototype translator application, we performed a formative evaluation to elicit how people interact with the technology and adapt their conversation style. We conducted two sets of studies with a total of 23 pairs (46 participants). Participants worked on storytelling tasks to simulate natural conversations with 3 different interface settings. Our findings show that collocutors naturally adapt their style of speech production and comprehension to compensate for inadequacies in SLT. We conclude the paper with the design guidelines that emerged from the analysis.

1. Kotaro Hara and Shamsi Iqbal
makeability lab
Effect of Machine Translation in Interlingual Conversation

2. こんにちは！

3. こんにちは！

4. Hello!

5. Successful intercultural communication is important in:
Education
Business
Health-care
Yet, a language barrier can be a challenge

6. “133/2011 euroscola” by rosipaw

7. Simultaneous interpretation is only for privileged few

8. “The infrequent use of
interpreters* in the
delivery ward was
among the most
important reasons for
the reduced quality of
[health] care.”
Kale and Syed (2010)
* Interpreter: a person who translates
the words that someone is speaking
into a different language
(Merriam-Webster)

9. Automatic Spoken Language Translation

11. Speech Recognition & Machine Translation
Performance
Fügen et al. (2007) noted that “automatic systems can
already provide usable information for people.”
2015

12. 2015
Little research has paid attention to how people interact with
spoken language translation technology

13. Very little knowledge
about Spoken Language
Translation in HCI
“Jigsaw puzzle” by James Petts

14. The void that we are trying to fill
Very little knowledge
about Spoken Language
Translation in HCI

15. Background
背景

16. Hamon O., et al., End-to-End Evaluation in Simultaneous Translation, ECACL2009
Hello!
Speech
Text
Spoken Language Translation System
Spoken Language Translation
Automatic speech recognition Machine translation
Speech synthesis
こんにちは
a
あ

17. Hamon O., et al., End-to-End Evaluation in Simultaneous Translation, ECACL2009
Hello!
Speech
Text
Spoken Language Translation System
Spoken Language Translation
Automatic speech recognition Machine translation
Speech synthesis
こんにちは
a
あ

18. Danger.
Do not enter
Effect of machine translation on text-based communication
Gao et al. 2014; Yamashita et al. 2009; Yamashita and Ishida 2006

19. Hello!
Native English SpeakerNon-native English Speaker
Hello!
Effect of speech recognition on interlingual conversation
Gao et al. CHI2014

20. Key Point
Speech
Recognition Text Output
Machine
Translation

21. Effect of spoken language translation system to
interlingual communication is under-explored.
Key Point

22. Evaluation of NESPOLE!
Constantini et al. 2002
• No detailed discussion on
how people adapted in
using the system
• Push-to-talk interface made
it hard to analyze turn-taking
behavior
• Examined usability of
spoken language
translation system

23. Explore how spoken language translation
affects a natural conversation between
people speaking different languages
Goal

24. Translator
Tool
Study
Method
Quantitative
Analysis
Content
Analysis

25. Translator Tool

26. Video to show how the system work:
One side speaks in English, another
side in German
Translator Tool | Demo
Skype Translator Demo, English-German Conversation

27. Video to show how the system work:
One side speaks in English, another
side in German
Translator Tool | Demo

28. Video to show how the system work:
One side speaks in English, another
side in German
Video chat interface
Translator Tool | Interface Components

29. Video to show how the system work:
One side speaks in English, another
side in German
Closed Caption (CC)
Translator Tool | Interface Components

30. Video to show how the system work:
One side speaks in English, another
side in German
Translated Text-to-Speech (TTS) Audio
Translator Tool | Interface Components

31. Video to show how the system work:
One side speaks in English, another
side in German
Headset
Microphone
Translator Tool | Interface Components

32. Translator
Tool
Study
Method
Quantitative
Analysis
Content
Analysis

33. Translator
Tool
Study
Method
Quantitative
Analysis
Content
Analysis

34. Study Method | Participants
8 French-German pairs
(N=16 participants)
15 English-German pairs
(N=30 participants)

35. Study Method | Participants
8 French-German pairs
(N=16 participants)
15 English-German pairs
(N=30 participants)
No common language With common language
(English)
All participants could speak
English reasonably well.

36. Study Method | Participants
8 French-German pairs
(N=16 participants)
15 English-German pairs
(N=30 participants)
With common language
(English)
I will mainly talk about
this study
No common language

37. Study Method | Conversation Task
Olivia trainierte für den
Tanzwettbewerb.
(Olivia was practicing for the
dance-off)
Conversation Task
German Speaker French Speaker

38. A starting sentence was
provided to the participant
Study Method | Conversation Task
Olivia trainierte für den
Tanzwettbewerb.
(Olivia was practicing for the
dance-off)
Conversation Task
German Speaker French Speaker
Translate

39. Study Method | Conversation Task
Olivia trainierte für den
Tanzwettbewerb.
(Olivia was practicing for the
dance-off)
Je ne savais pas Olivia dansé.
Combien de temps elle s'y
adonne ?
(I didn’t know Olivia danced. How long
has she been practicing?)
Conversation Task
German Speaker
Translate
French Speaker

40. Study Method | Conversation Task
Olivia trainierte für den
Tanzwettbewerb.
(Olivia was practicing for the
dance-off)
Seit sechs Jahren
(For six years)
C'est une longue période.
(That’s a long time.)
Conversation Task
German Speaker
…
French Speaker
Je ne savais pas Olivia dansé.
Combien de temps elle s'y
adonne ?
(I didn’t know Olivia danced. How long
has she been practicing?)

41. We asked participants to perform 9 conversation tasks
(~3 min ea.) in their respective languages and an
additional conversation task in English
The goal of each task was to collaboratively construct
a coherent story
Study Method | Conversation Task
Tasks were conducted using three different settings of
the translator tool
Conversation Task

42. Study Method | Interface Settings
Closed Caption &
Text-to-Speech (CC & TTS)
CC
Closed Caption only (CC)
CC
Text-to-Speech only (TTS)
Interface Settings

43. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech
With Closed Caption and Text-to-Speech
CC CC

44. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech
CC CC

45. Closed Caption only
Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech
CC CC

46. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech
CC CC

47. Text-to-Speech only
Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech
CC CC

48. Closed Caption & Text-to-SpeechClosed Caption Text-to-Speech
CC CC

49. Closed Caption and
Text-to-Speech
CC
Closed Caption
CC
Text-to-Speech
Round 1
Study Method | Interface Settings
Interface Settings
Order was permuted for each pair
for counter balancing

50. CC CCRound 1
Study Method | Interface Settings
Interface Settings
CC CC
CC CC
Round 2
Round 3
9 tasks with different interface settings and
different starting sentences

51. CC CCRound 1
Study Method | Interface Settings
Interface Settings
CC CC
CC CC
Round 2
Round 3
Eng A baseline English task to compare
with/without translation settings

52. CC CCRound 1
Study Method | Data
Data
CC CC
CC CC
Round 2
Round 3
Post-task survey (after every task)
E.g., “We had a successful conversation.”
Strongly
Agree
Strongly
Disagree
Eng

53. Eng
CC CCRound 1
Study Method | Data
Data
CC CC
CC CC
Round 2
Round 3
Post-round survey
(after every 3 tasks)
Interface setting preference ranking
CC & TTS: __________, CC: __________, TTS: __________Best Neutral Worst

54. CC CCRound 1
Study Method | Data
Data
CC CC
CC CC
Round 2
Round 3
Eng Post-session survey
and Interview

55. Translator
Tool
Study
Method
Quantitative
Analysis
Content
Analysis

56. Translator
Tool
Study
Method
Quantitative
Analysis
Content
Analysis
• Overall interface
setting preferences
• Whether people became
used to using the
translator tool

57. Three interface settings
Three rounds (nine tasks)
Two languages
3 x 3 x 2 mixed design study
With-in subject
With-in subject
Between subject
Quantitative Analysis | Analysis Method
Analysis Method

58. Transformed ordinal data in survey responses with
aligned rank transformation
We analyzed the data with restricted maximum
likelihood model
Quantitative Analysis | Analysis Method
Analysis Method

59. Interface Preference Results
Quantitative Analysis | Interface Preference Results
Which interface setting did you favor the most and least?
Closed Caption & Text-to-SpeechClosed Caption only Text-to-Speech only
CC CC

60. 59%
35%
6%
0%
25%
50%
75%
100%
Closed Caption & Text-to-
Speech
Closed Caption only Text-to-Speech only
Quantitative Analysis | Interface Preference Results
Most Favored Interface Setting
by French-German Group
CC CC
Interface Settings
Percentage of people who
favored the interface setting

61. 59%
35%
6%
0%
25%
50%
75%
100%
Closed Caption & Text-to-
Speech
Closed Caption only Text-to-Speech only
Quantitative Analysis | Interface Preference Results
Most Favored Interface Setting
by French-German Group
Interface settings with closed
caption were preferred
CC CC

62. 59%
35%
6%
0%
25%
50%
75%
100%
Closed Caption & Text-to-
Speech
Closed Caption only Text-to-Speech only
Quantitative Analysis | Interface Preference Results
Most Favored Interface Setting
by French-German Group
CC CC
24% more people preferred
to have text-to-speech

63. 59%
35%
6%
0%
25%
50%
75%
100%
Closed Caption & Text-to-
Speech
Closed Caption only Text-to-Speech only
Quantitative Analysis | Interface Preference Results
Most Favored Interface Setting
by French-German Group
CC CC
Set the Closed Caption & Text-to-Speech to the default,
but allow users to turn on/off setting

64. 2.2 2.2
2.7
4.8
1
2
3
4
5
Round 1 Round 2 Round 3 English
Overall Conversation Quailty
Perceived Conversation Quality over Rounds
Round
Average 5-point
Likert scale rating
Quantitative Analysis | Perceived Conversation Quality
5-point Likert scale rating (higher is better)
Rating

65. 2.2 2.2
2.7
4.8
1
2
3
4
5
Round 1 Round 2 Round 3 English
Overall Conversation Quailty
p < 0.01
p < 0.01
Participants felt that
their conversation
quality improved
Perceived Conversation Quality over Rounds
Quantitative Analysis | Perceived Conversation Quality
F2,112=6.275, p<0.01; Error bars are standard errors
5-point Likert scale score (higher is better)
Rating

66. 2.2 2.2
2.7
4.8
1
2
3
4
5
Round 1 Round 2 Round 3 English
Overall Conversation Quailty
Perceived conversation
quality is not as good as
that of English task
Perceived Conversation Quality over Rounds
Quantitative Analysis | Perceived Conversation Quality
5-point Likert scale score (higher is better)
Rating

67. Translator
Tool
Study
Method
Quantitative
Analysis
Content
Analysis

68. Translator
Tool
Study
Method
Quantitative
Analysis
Content
Analysis
• Why people were satisfied or
dissatisfied with the experience
• How people adapted to using
the translator tool

69. Interview transcripts and survey responses were
coded with a content analysis method; recurring
themes were extracted and grouped together
Post-session interviews were transcribed by authors
Content Analysis | Method
Method

70. 62.5%
of the participants noted that
proper noun recognition
errors hindered the
conversation.
Content Analysis | Proper Noun Recognition Errors
Proper Noun Recognition Errors

71. ‘Ava’, there was no way it was getting ‘Ava’ no matter how
many times I tried. So in German, when you say ‘but,’ it
sounds very similar. So that’s what it was picking up.
French-German Pair 4, German speaker
Content Analysis | Proper Noun Recognition Errors

72. ‘Ava’, there was no way it was getting ‘Ava’ no matter how
many times I tried. So in German, when you say ‘but,’ it
sounds very similar. So that’s what it was picking up.
French-German Pair 4, German speaker
Content Analysis | Proper Noun Recognition Errors
Lack of an adaptation technique

73. Content Analysis | Grammar and Word Order Errors
37.5%
of the participants noted that
grammatical errors and
misplacement of words were
problematic.
Grammar and Word Order Errors

74. French-German Pair 3, German speaker
Because [...] German sentence structure is so different,
[translated] sentence wouldn’t make any sense when
[synthesized speech] said it, but when you see it on the
screen, you can kind of reorganize the words in the way it
supposed to go. And then it makes sense.
Content Analysis | Grammar and Word Order Errors

75. French-German Pair 3, German speaker
Because [...] German sentence structure is so different
[translated] sentence wouldn’t make any sense when
[synthesized speech] said it, but when you see it on the
screen, you can kind of reorganize the words in the way it
supposed to go. And then it makes sense.
Content Analysis | Grammar and Word Order Errors
People could find a
fallback strategy

76. Content Analysis | Information from Original Speech
31.3%
of the participants noted that
original speech was useful.
Information from Original Speech

77. [Having partner’s original voice] humanizes the
relationships as well, because if you only get the robotic
voice, I think it just cuts you off from the person you are
engaged with. Whereas you talk, then it feels much more
human relationship.
French-German Pair 1, French speaker
Content Analysis | Information from Original Speech
An original speech can be useful
even when a person do not know
the language

78. Content Analysis | Forgiving
31.3%
of the participants mentioned
that they forgave the errors of
the translator tool
Forgiving

79. My sentence became shorter, I pronounced more clearly,
I was speaking ... I mean, if I were speaking to someone
who’s not a native German, and if I didn’t have a
translator, then I would also slow down, obviously.
French-German Pair 5, German speaker
Content Analysis | Forgiving

80. French-German Group English-German Group
Difference in Language Groups

81. My partner and I would begin to speak then the
translation from the previous statement would catch up,
translation and speaker would then all be talking at the
same time.
English-German Pair 4, English speaker
Content Analysis | Speech Overlap

82. Content Analysis | Speech Overlap
Speech Overlap
25.0%
of the participants mentioned
that they speech overlap was
a problem
French-German Group

83. Content Analysis | Speech Overlap
Speech Overlap
25.0%
50.0%
Participants from the English-German group perceived
speech overlap as a more severe problem
French-German Group English-German Group
This was partly because German
speakers could respond without
seeing/hearing translation.

84. Design Guidelines

85. Design Guidelines
• Support Users’ Adaptation for Speech Production
• Offer Fallback Strategies with Non-verbal Input
• Support Comprehension of Messages
• Support Users’ Turn Taking

86. Limitation German Speakers’ English Proficiency

87. Limitation
Interpreter
Baseline Task

88. Hi!
안녕!
Future Work
Analysis of conversation
Conduct follow up
summative studies

89. Summary
We used Skype Translator to elicit interaction
problems in using spoken language translation for
interlingual conversation.
Our analysis revealed how people adjusted
(or could not adjust) their behaviors to overcome
the problems.
makeability lab

90. Kotaro Hara Shamsi T. Iqbal
Microsoft Skype Translator team
makeability lab

91. @kotarohara_en
Questions?
makeability lab

92. makeability lab
Image and Icon Credit
- Friendlies by Mo Riza (https://www.flickr.com/photos/moriza/2565606353)
- 133/2011 euroscola by rosipaw (https://www.flickr.com/photos/rosipaw/5768035647/)
- Japantimes (http://www.japantimes.co.jp/news/2014/05/08/world/social-issues-world/births-fall-economies-may-falter)
- Windows8core.com (http://www.windows8core.com/bing-translator-app-for-windows-8rt-lands-in-store-review/)
- Jigsaw puzzle by James Petts
- Speech bubble from Wikimedia Commons (http://commons.wikimedia.org/wiki/File:Speech_bubble.svg)
- Head silhouette from Channelweb (http://www.channelweb.co.uk/crn-uk/news/2244637/good-week-bad-week)
- Microphone from Uxrepo.com (http://uxrepo.com/icon/mic-by-mfg-labs)
- Document from IconArchive (http://www.iconarchive.com/show/mono-general-2-icons-by-custom-icon-design/document-icon.html)
- Funny Signs from Fanpop (http://www.fanpop.com/clubs/picks/images/1771904/title/funny-signs-photo)
- Compass Zoomed by Gwgs (https://www.flickr.com/photos/gwgs/346806882)
- Beaker by Emily van den Heever (https://thenounproject.com/EmvdHeever/)