MT providers claim that (customized) MT "helps you translate more words and grow your business". It "boosts productivity". And it can even "increase revenues" or "optimize customer service and support". What is the reality? Is MT improving or did we reach a plateau? Speakers at last year's QE Summit agreed: one of the main problems in the translation industry today is the lack of benchmarking. The output of MT engines cannot be compared to industry averages or standards because these are not yet available. Automated scores are meaningless outside the “laboratory”. At the same time, buyers of translation services are increasingly interested in translated content of different quality levels. They also want to know how the different engines are performing on different content types and in different language pairs. How do we know? Can we predict the output quality? Shouldn’t MT providers become more transparent to help buyers of these technologies make informed decisions? Session leader: Dag Schmidtke (Microsoft) Panelists: John Tinsley (Iconic), Olga Beregovaya (Welocalize), Olga Pospelova (eBay)

1. Machine Translation at eBay: Are We There Yet?
Olga Pospelova 2016

2. • eBay serves over 7 billion translations daily
• e-commerce domain systems for 10+ language directions
• Content: item titles, item descriptions, user search
queries, user-to-user communication

3. From Raw Machine Translation to Human Quality
MT
output
Human
Translatio
n
Light Post-Editing Full Post-Editing
Syntax
Errors
eBay
Specific
Requirements
Minor
Gramma
r Errors
Fluency
/Style
Lexica
l
Errors
Improving MT Output: High Quality Training Corpus

4. Quality Management
Set quality expectations
Pilot projects
Training and onboarding
Context
Reference materials
TMs
Automatic sampling of
content for review
Provide continuous feedback
to post-editors/translators
Vendor
Management

5. Improving MT Output: NER tagging
Brand
name
Product
name
Storage
capacity
Color
Brand
name

6. Human Evaluation
Source Target Source
segment OK?
All of
information
preserved?
Main noun
identifying
item translated
correctly?
Translation
alone easy to
read and
understand?
Brand names
correctly
translated?
new apple (iwatch)
watch sport edition
42mm - pre-order -
worldwide ship!
novo apple iwatch,
relógio , edição de
esporte 42mm-pré-
encomenda - mundial
navio!
Y Y N Y
birkenstock madrid
mujer castao claro
estrechos sandalias
zapatos eu 40
birkenstock madrid
mujer castao claro
estrechos sandalias
sapatos da ue 40
N
• Rating (1-5 scale)
• Ranking
• Error Analysis
• Decision Tree

7. User Evaluation