In this session, with clear focus on Machine Translation (MT) quality, we will discuss different ways to improve MT engines. Which engine do you use and how do you measure improvement? What are the right metrics to evaluate MT quality for the specific content types? How do you interpret and act on the evaluation results? It's fine when errors are labeled and analyzed, but how can that help improve your engine? Are there best practices available? And how about Neural MT? Should we measure that differently? After some use cases shared by the speakers, these questions will be addressed in the break-out session.

1. CONFIDENCE IN OUR QUALITY ESTIMATION

2. Objectives
How confident are we on our QE?
Can we map the QE to a prediction of usability of output?
What are the different deployment models?
Deployment of QE scores for our
Manufacturing engines

3. Model
Feature Extraction ClassificationMT output QE SCORE
Quality Estimation
Human
Evaluation
MT output Score
Mapping
CONFIDENCE
RATING
Validation

4. Results for our Manufacturing Engines
EN->ES
DE->EN
EN->DE
EN->FR
EN->PL
EN->NL
Languages
• QE Range 0-1 reverse
• H Score 1-4
• 100% 0 to 0.3 scores were H Score 4
• 55% of QE scores of 0.8 or higher were
over 3 H Score
• 33% of H Score 4 were scored over 0.5
QE Score H Score
0-0.3 4
0.4 – 0.5 3.5
0.6-0.7 2.7
0.8-1 2.9

5. Deployment models
QE Score
Segment
Output
QE Score
Productivity
Content/Engine
suitability
Segment quality
Productivity prediction
based on scoring