Workshop orientado por Lucia Specia na I Conferência Internacional de Tradução e Tecnologia, 13 e 14 de Maio, Faculdade de Letras do Porto.

1. Statistical Machine Translation and
Post-editing
I Conferˆencia Internacional de Tradu¸c˜ao e Tecnologia
Porto, 14 Maio 2013
Lucia Specia
(Thanks to Wilker Aziz for many of the SMT slides)
Department of Computer Science
University of Sheffield
http://www.dcs.shef.ac.uk/~lucia/
Porto (14/05/2013) SMT and PE 1 / 67

2. Overview
• Machine Translation - task and challenges
• Statistical Machine Translation
Word-based models
Phrase-based models
Syntax-based models
• Post-editing:
Process and research
Practice with PET
Porto (14/05/2013) SMT and PE 2 / 67

3. The task of Machine Translation (MT)
The boy ate an apple
O menino comeu uma ma¸ca
BUT
He said that the bottle floated into the cave
? Dijo que la botella entro a la cueva flotando
虽然 北 风 呼啸 , 但 天空 依然 十分 清澈 。
However , the sky remained clear under the strong north wind .
Porto (14/05/2013) SMT and PE 3 / 67

4. Challenges in MT: Ambiguity
• Lexical ambiguity: different words meanings may translate differently
e.g. book the flight ⇒ reservar
read the book ⇒ libro
e.g. kill a man ⇒ matar
kill a process ⇒ acabar
• Syntactic ambiguity:
e.g. John hit the dog with the stick
⇒ John golpeo el perro con el palo
⇒ John golpeo el perro que tenia el palo
• Pronoun resolution:
e.g. The computer outputs the data; it is fast
⇒ La computadora imprime los datos; es rapida VS.
e.g. The computer outputs the data; it is stored in ascii
⇒ La computadora imprime los datos; estan almacendos en ascii
Porto (14/05/2013) SMT and PE 4 / 67

5. Challenges in MT: Divergences
• Structural divergences:
e.g. The bottle floated into the cave
⇒ La botella entro a la cueva flotando
(The bottle entered the cave floating)
• Different word orders:
English is: subject – verb – object
Japanese is: subject – object – verb
English: IBM bought Lotus yesterday
Japanese: IBM Lotus bought yesterday
Porto (14/05/2013) SMT and PE 5 / 67

6. Challenges: Non-literal meaning
• Idioms
e.g. He finally kicked the bucket at the hospital
⇒ Ele finalmente chutou o balde no hospital VS.
⇒ Ele finalmente bateu as botas no hospital
• Multi-word expressions
e.g. Do take the long waiting list for organ donation in this country into
account
⇒ Considere a longa lista de espera para doa¸c˜ao de ´org˜aos neste pa´ıs
Porto (14/05/2013) SMT and PE 6 / 67

7. Requires deep language processing
• One of the most challenging problems in NLP
• Requires
Full understanding of source text (analysis)
Generation of a coherent, accurate, fluent target text (synthesis)
Porto (14/05/2013) SMT and PE 7 / 67

8. Statistical Machine Translation
P(French|English)
• Statistical Machine Translation (SMT) learn how to generate
translations using statistical methods
• Early 1990s by IBM. Adopted by Google, etc.
• Idea is older:
Warren Weaver (1949)
When I look at an article in Russian, I say: “This is really written in
English, but it has been coded in some strange symbols. I will now
proceed to decode.”
Inspired by WWII code-breaking
Approach not feasible with early computers – idea not pursued
Porto (14/05/2013) SMT and PE 8 / 67

9. Probabilistic Model
• Output of decoder depends probabilistically on the input
• To translate French (F) into English (E):
Given a French sentence F search for English sentence E∗ that maximises
P(E|F)
Porto (14/05/2013) SMT and PE 9 / 67

10. Probabilistic Model
• Find English sentence that maximises P(E|F), i.e.
E∗ = argmax
E
P(E|F)
= argmax
E
P(E)·P(F|E)
P(F) Bayes Rule
P(F) constant across different E, so:
E∗ = argmax
E
P(E) · P(F|E)
drop constant
divisor P(F)
Porto (14/05/2013) SMT and PE 10 / 67

11. Noisy Channel Model & SMT (ctd)
E∗
= argmax
E
P(E|F) = argmax
E
P(E) · P(F|E)
• Why not model P(E|F) directly?
Would need very good probability estimates
Divide and conquer!
• Decomposition to P(E) · P(F|E) allows one to use less reliable
probabilities
P(E) worries about good English — fluency
• ensures that E has words in the right order and fit together
P(F|E) worries about French that translates English — faithfulness
• ensures that E has words that generally translate F
P(E) and P(F|E) can be trained independently
Porto (14/05/2013) SMT and PE 11 / 67

12. Main Components for Translation (F → E)
• Translation model (TM): P(F|E)
Faithfulness: TMs learned by analysing large amounts of parallel text
• Judge which (F, E) pairs are likely to be translations of each other
• Language model (LM): P(E)
Fluency: LMs created from large volume of (fluent) target language
sentences
• Learn which word sequences are common/more probable in E
• A Decoder: (argmax)
Search algorithm to find E∗
• V. large search space — heuristic search approach used
Porto (14/05/2013) SMT and PE 12 / 67

13. Learning Translation Models - P(F|E)
• Requires a bilingual corpus
e.g. European/Canadian/Hong Kong parliaments, subtitles, Bible
• Bilingual corpus must be sentence-aligned:
• Can we estimate P(F|E) from whole sentences?
Porto (14/05/2013) SMT and PE 13 / 67

14. Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
Porto (14/05/2013) SMT and PE 14 / 67

15. Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
• The same word happens in source 1 and 3
English
A home
A garden
I arrived home
Porto (14/05/2013) SMT and PE 15 / 67

16. Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
• Could we expect the same in the target side?
English
A home
A garden
I arrived home
Porto (14/05/2013) SMT and PE 16 / 67

17. Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
• useh = home
Porto (14/05/2013) SMT and PE 17 / 67

18. Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
• What about the contexts?
English
A home
A garden
I arrived home
Porto (14/05/2013) SMT and PE 18 / 67

19. Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
• We can align them: Yiguo = I; Yiguo = arrived; Uh = A
Porto (14/05/2013) SMT and PE 19 / 67

20. Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
• And reuse this knowledge to align more sentences: Uh = A
Porto (14/05/2013) SMT and PE 20 / 67

21. Learning Translation Models - Word-based SMT
• Break sentences into smaller units: words are a good starting point
• Learn translation probabilities by word aligning a sentence-aligned
corpus:
Zenish
Uh useh
Uh jejje
Yiguo useh
English
A home
A garden
I arrived home
• And again: jejje = garden
Porto (14/05/2013) SMT and PE 21 / 67

22. Learning Translation Models - Word-based SMT (ctd)
Word-alignment:
• Identify correspondences between two languages at the word level
• Basis for word-based SMT, first step for other approaches
• Alignment can be learned using Expectation Maximization (EM)
Treat all alternative word alignments as equally likely
Observe across sentences that Zenish useh often links to English home
• Increase probability of this word pair aligning
• Has knock-on effect for alignment of other words
Iteratively redistribute probabilities, until they identify most probable
links for each word (convergence)
Word alignment commonly done using the IBM Models 1-5
Porto (14/05/2013) SMT and PE 22 / 67

23. Learning Translation Models - Word-based SMT (ctd)
• IBM models produce a probabilistic dictionary for the whole parallel
corpus:
contras contras 0.0285714
contras los 0.0000011
contras esos 0.0000812
richer enriquecida 0.0476190
richer mucho 0.0000502
richer enriqueci 0.0769231
richer variado 0.0066225
Porto (14/05/2013) SMT and PE 23 / 67

24. Learning Translation Models - Word-based SMT (ctd)
Back to the translation model:
At translation (decoding time), for a new sentence to translate, take the
set of translations that jointly maximise the whole translation probability
E∗ = argmax
E
P(F|E)
• Generates alternative translations using different translation options
for words and different word orders
• Selects the one with maximum score: no notion of context. How
about ambiguity?
What about the context and the fluency in the target language?
Porto (14/05/2013) SMT and PE 24 / 67

25. Learning Language Models - Word-based SMT
Language model: P(E)
E∗ = argmax
E
P(F|E) · P(E)
• Takes different translation options, tests how “common” each of
them are in the target language as a whole sentence (some context)
• P(E) = probability of strings E based on relative frequencies in a
large corpus of language E
Google
the house is small 144,000
the house is little 85,300
small house 2,330,000
little house 3,330,000
Porto (14/05/2013) SMT and PE 25 / 67

26. Learning Language Models - Word-based SMT (ctd)
• Model P(E) by decomposing into single-word probabilities given
previous words, under Markov assumption: only the previous n-1
words matter for predicting a word. E.g for trigram models, n = 3
P(e1) · P(e2|e1) · P(e3|e1, e2) · P(e4|e2, e3) · · · P(en|en−2, en−1)
• Relative frequencies to compute these probabilities. E.g. trigrams:
P(e3|e1, e2) = count(e1e2e3)
count(e1e2)
P(little|house, is) = count(house is little)
count(house is)
Porto (14/05/2013) SMT and PE 26 / 67

27. Word-based SMT – Limitations
• It is difficult to do alignment, and hence learn a TM, for languages
that are structurally quite different, with different words orders
Considering all possible word orders – too costly
Poor reordering model: flexibility vs noise
• Some languages may have different notions of what counts as a word
Donaydampfshiffahrtsgesellschaftskapitaenskajuetenschluesseloch
The keyhole of the door of the cabin of the captain of a steamship
company operating on the Danubea
a
http://www.chebucto.ns.ca/~waterbuf/longwrds.html
Porto (14/05/2013) SMT and PE 27 / 67

28. Phrase-based SMT
• Leading approach since 2003
No voy1 a la2 casa3 → I am not going1 to the2 house3
it seems to1 me2 → me1 parece2
Je1 ne vais pas2 `a la3 maison4 → I1 am not going2 to the3 house4
Eu1 sinto saudade de vocˆe2 → I1 miss you2
I1 miss you2 → Eu1 sinto sua falta2
natuerlich1 hat2 John3 spass am4 spiel5 → of course1 John2 has3 fun with the4 game5
• More intuitive and reliable alignments
Account for reordering within the phrases
Phrases can still be reordered
• Can we learn phrase translation probability distribution from the
parallel corpus using EM?
1 src-tgt sentence → O(n4
) phrase pairs
Porto (14/05/2013) SMT and PE 28 / 67

29. Phrase-based SMT - Phrases from word alignments
• Extract phrase pairs that are consistent with the word alignment
• Phrase: sequence of tokens, not linguistically motivated
• Word alignment produced by any of the IBM Models - only done once
reanudaci´on del per´ıodo de sesiones
resumption
of
the
session
1 resumption ↔ reanudaci´on
2 of the ↔ del
3 session ↔ per´ıodo de sesiones
4 resumption of the ↔ reanudaci´on del
5 of the session ↔ del sesiones
6 resumption of the session ↔ reanudaci´on del sesiones
Porto (14/05/2013) SMT and PE 29 / 67

30. Phrase-based SMT - Computing phrase probability
distributions
1 Extract phrase pairs from word aligned parallel corpus
√
2 Extract counts of those phrases from large parallel corpus:
φ(¯f |¯e) =
count(¯f , ¯e)
count(¯e)
3 Store phrases and their probabilities in a phrase table
Probabilistic dictionary of phrases
Porto (14/05/2013) SMT and PE 30 / 67

31. Phrase-based SMT - Limitation of Noisy Channel Model
E∗ = argmax
E
P(F|E) · P(E)
• Which component is more important?
P(F|E) ?
P(E) ?
• Depends on the corpus used (more parallel data makes P(F|E) more
reliable), language-pair, etc.
Porto (14/05/2013) SMT and PE 31 / 67

32. Phrase-based SMT - Log-linear Model (ctd)
Assign weights λi to components (feature functions) hi (f, e):
e∗ = argmax
e
n
i=1 λi hi (f, e)
Components
1 P(E)
2 P(F|E)
Weights
1 λP(E)
2 λP(F|E)
Feature Functions
1 h1 = logP(E)
2 h2 = logP(F|E)
Benefits
1 Extensible: other components are used: e.g. reordering, word and
phrase penalties
2 Weights can be tuned, i.e., learned from examples to minimise error
in a subset of the parallel corpus, e.g. MERT
Porto (14/05/2013) SMT and PE 32 / 67

33. Phrase-based SMT - Decoding (ctd)
Search problem: segmenting source sentence and selecting phrase
translations:
• All phrases matching source words are selected from the phrase table
J’ ai les yeux noirs .
I have the eyes black .
me has them eye dark ,
I have eyes espresso !
I am the eyes somber .
I did some black eyes .
I had black eyes .
I have black eyes .
black eyes I have .
• The goal of the decoder is to select the phrases whose combination
(in a given order) yields the highest score for the log-linear model
Porto (14/05/2013) SMT and PE 33 / 67

34. Phrase-based SMT - Decoding (ctd)
Searching = incrementally construct the translation hypotheses by trying
out several possibilities
• Generating target words in sequence, from left to right
• Incrementally compute the overall log-linear model score for each
hypothesis
• Heuristics to prune search space, e.g. stack-based beam search
Use one stack for of every length of partial hypotheses
Limited space for partial hypotheses, only keep those that are
promising, e.g. model score is close to that of the best partial
hypothesis so far
Porto (14/05/2013) SMT and PE 34 / 67

35. Phrase-based SMT - Decoding (ctd)
Search space
Hypothesis
• Covered source words
• Target (output) words
• Model score (using the combined probabilities)
Porto (14/05/2013) SMT and PE 35 / 67

36. Hierarchical and Syntax-based SMT
• PBSMT still has trouble with long-distance reorderings
• Ways of bringing structure and linguistic knowledge into the transfer
rules of the phrase table
Porto (14/05/2013) SMT and PE 36 / 67

37. Hierarchical SMT - Motivation
Introduce structure into phrase-based SMT models to deal with
long-distance reordering
Ich
werde
Ihnen
die
entsprechenden
Anmerkungen
aush¨andigen
I
shall
be
passing
on
to
you
some
comments
Porto (14/05/2013) SMT and PE 37 / 67

38. Hierarchical SMT - Motivation
Introduce structure into phrase-based SMT models to deal with
long-distance reordering
Ich
werde
Ihnen
die
entsprechenden
Anmerkungen
aush¨andigen
I
shall
be
passing
on
to
you
some
comments
• How can we get a phrase for shall
be passing on?
Porto (14/05/2013) SMT and PE 38 / 67

39. Hierarchical SMT - Motivation
Introduce structure into phrase-based SMT models to deal with
long-distance reordering
Ich
werde
Ihnen
die
entsprechenden
Anmerkungen
aush¨andigen
I
shall
be
passing
on
to X
you X
some X
comments X
• How can we get a phrase for shall
be passing on?
• We cannot, unless we get to you
some comments along
Porto (14/05/2013) SMT and PE 39 / 67

40. Hierarchical SMT - Motivation
Introduce structure into phrase-based SMT models to deal with
long-distance reordering
Ich
werde
Ihnen
die
entsprechenden
Anmerkungen
aush¨andigen
I
shall
be
passing
on
to
you
some
comments
• How can we get a phrase for shall
be passing on?
• We cannot, unless we get to you
some comments along
• Unless we replace all those words
by a variable
Porto (14/05/2013) SMT and PE 40 / 67

41. Hierarchical SMT - Motivation (ctd)
shall be passing on to you some comments
werde Ihnen die entsprechenden Anmerkungen aush¨andigen
shall be passing on to you some comments
werde Ihnen die entsprechenden Anmerkungen aush¨andigen
shall be passing on X
werde X aush¨andigen
Porto (14/05/2013) SMT and PE 41 / 67

42. Hierarchical SMT - basics
Learnt from word-aligned parallel corpora in the same way as before
• Based on the fact that language has recursive structures
• Phrases within other phrases treated as nonterminals: replaced by X
• No linguistic constraints added - yet, some structure
Porto (14/05/2013) SMT and PE 42 / 67

43. Hierarchical SMT - basics (ctd)
shall be passing on to you some comments
werde Ihnen die entsprechenden Anmerkungen aush¨andigen
shall be passing on X1 some comments
werde X1 die entsprechenden Anmerkungen aush¨andigen
shall be passing on X1 X2
werde X1 X2 aush¨andigen
Porto (14/05/2013) SMT and PE 43 / 67

44. Hierarchical SMT - phrase-table
[X] → shall be passing on X1 X2 | werde X1 X2 aush¨andigen
[X] → shall be passing on X3 | werde X3 aush¨andigen
[X] → to you | Ihnen
[X] → some comments | die entsprechenden Anmerkungen
[X] → to you some comments | Ihnen die entsprechenden Anmerkungen
Learn a bilingual (synchronous) set of context-free rules about how to
translate F into E
Porto (14/05/2013) SMT and PE 44 / 67

45. Syntax-based SMT
• Hierarchical models are not very informative (Xs) and suffer from an
exponential number of rules
• Syntax-based SMT overcomes these limitations by using linguistic
categorise to label nodes
• A syntactic parser is required at pre-processing time on at least one
language (the other could have Xs)
• Idea:
Learn a bilingual (synchronous) set of linguistically informed
context-free rules about how to translate F into E
Rules are learnt from the word-alignment, constrained by syntactic
categories
Porto (14/05/2013) SMT and PE 45 / 67

46. Syntax-based SMT (ctd)
S
PRP
J’
VP
VB
ai
NP
DT
les
NN
yeux
JJ
noirs
S
PRP
I
VP
VB
have
NP
JJ
black
NN
eyes
J’ ai les yeux noirs
I
have
black
eyes
Standard constraints on rule construction:
• Single nonterminal on the left
• Consistent with word-alignment
• Nonterminals on the right must align
one-to-one
Porto (14/05/2013) SMT and PE 46 / 67

47. Syntax-based SMT (ctd)
Grammar
PRP → J’ | I
JJ → noirs | black
NP → les yeux JJ | JJ eyes
VP → ai NP | have NP
S → PRP VP | PRP VP
Decoding becomes a (probabilistic) parsing problem!
Porto (14/05/2013) SMT and PE 47 / 67

48. Summary on SMT
• Adding linguistic knowledge to SMT is a trend:
Stronger theoretical motivation
Deals with reordering, sparsity, ambiguity
Depends on (quality of) parsers
Higher complexity
• Performance:
Hard to quantify in absolute terms
Most language pairs → phrase-based or hierarchical SMT does better
Language pairs with long-distance reordering → syntax-based SMT
does better
• Other open issues:
Limited use of linguistic information - at most syntax, some attempts
to use shallow semantics - not yet there
Limited use of context - sentence-level
Domain discrepancies - mistranslations, unknown words
• Overall, to reach publishable quality, human checking/revision
necessary: post-editing
Porto (14/05/2013) SMT and PE 48 / 67

49. Definitions
Note: Some slides are based on those by Sharon O’Brien (tutorial on MT
post-editing at AMTA-2010).
What is post-editing?
• Same as translation?
• Same as revision?
• A new profession? New skills?
Who does the post-editing?
What are the quality expectations?
Porto (14/05/2013) SMT and PE 49 / 67

50. Degrees/types of post-editing
• Light (fast/rapid/gist) post-editing: essential corrections
• Full post-editing: more corrections, higher quality, closer to revision
• Monolingual post-editing: no access to source text
Decided by:
• User requirements
• Volume
• Quality expectations
• Turn-around time, etc.
Porto (14/05/2013) SMT and PE 50 / 67

51. Quality of post-editing
• Can MT+PE (possibly +revision) produce the same level of quality
as HT+revision?
• Dependent on the quality of MT systems: variable for language pair,
text type/genre & domain, quality of source
• Dependent on quality requirements
• Different outcomes:
Autodesk reports lower error rates with MT+PE [Aut11]
Tilde observed higher error rate with MT+PE [SPSV11] -
under-resourced highly-inflected languages: lower MT quality
Porto (14/05/2013) SMT and PE 51 / 67

52. Ways of measuring quality of MT for PE
• Automatic MT evaluation metrics ? BLEU/TER/METEOR... -
automatic, but do not correlate well with PE effort
• Error counts: compare source with raw MT output - manual, costly
Can we use the “standard” error typologies for HT? Do MT and HT
make same mistakes?
QTLaunchPad project
Multidimensional Quality Metrics for MT and HT, for manual and
(semi-)automatic evaluation (QE): http://www.qt21.eu/launchpad/
• % Changes made: compare post-edited text with raw MT output -
automatic
• Time to make changes: record time to fix raw MT - automatic
• Estimated effort: take source text and raw MT output and estimate
PE effort prior to post-editing (QE) - automatic, less precise
Is this the same as fuzzy match level in TM systems?
Porto (14/05/2013) SMT and PE 52 / 67

53. Aspects involved in post-editing effort
• Temporal: time to post-edit (variable across translators)
• Technical: no./% edits made, keys pressed
• Cognitive: time, pauses, movements
Studies on PE process:
• Is PEMT faster than HT? Yes
• Is PEMT less keyboard intensive than HT? Yes
• Is PEMT less cognitively demanding than HT?
Cognitive effort is the most difficult to estimate: active research area
[Kop12, KARS12]
• PE more tedious/tiring? Mixed opinions
Check work by
Michael Carl - CBS (Translog) and Sharon O’Brien - DCU (eye-tracking)
Porto (14/05/2013) SMT and PE 53 / 67

54. Pricing models
Open question. Some common practices:
• Paying as fuzzy segment matches (source-based)
• Paying as % of edits performed (MT&PE-based)
• Paying a fee based on time spent (less common)
Porto (14/05/2013) SMT and PE 54 / 67

55. PE Guidelines
• No standard guidelines, but good practices (e.g. TAUS/CNGL)
• Guidelines need to clearly specify level of post-editing and quality
expectations
• Guidelines need to clearly differentiate between essential and
preferential changes
• Ideally, guidelines should be system-, language-, purpose-specific
Porto (14/05/2013) SMT and PE 55 / 67

56. General PE Guidelines for “good enough” quality -
TAUS/CNGL
• Aim for semantically correct translation.
• Ensure that no information has been accidentally added or omitted.
• Edit any offensive, inappropriate or culturally unacceptable content.
• Use as much of the raw MT output as possible.
• Basic rules regarding spelling apply.
• No need to implement corrections that are of a stylistic nature only.
• No need to restructure sentences solely to improve the natural flow of
the text.
http://www.translationautomation.com/postediting/
machine-translation-post-editing-guidelines
Porto (14/05/2013) SMT and PE 56 / 67

57. General PE Guidelines for HT quality - TAUS/CNGL
• Aim for grammatically, syntactically and semantically correct
translation.
• Ensure that key terminology is correctly translated and that
untranslated terms belong to the clients list of Do Not Translate
terms.
• Ensure that no information has been accidentally added or omitted.
• Edit any offensive, inappropriate or culturally unacceptable content.
• Use as much of the raw MT output as possible.
• Basic rules regarding spelling, punctuation and hyphenation apply.
• Ensure that formatting is correct.
http://www.translationautomation.com/postediting/
machine-translation-post-editing-guidelines
Porto (14/05/2013) SMT and PE 57 / 67

58. IT PE Guidelines - Midori Tatsumi’s PhD thesis
• What needs to be fixed:
Non-translatable items, such as command and variable names, that
have been translated. Please put it back to English.
Inappropriately translated general IT terms.
Mistranslation.
Word orders that are inappropriate to the level that the sentence has
become impossible or difficult to comprehend.
Comprehensible but extremely unnatural or inappropriate expressions.
Inappropriate postpositions and conjugations.
• What are preferred to be standardised:
Repetitive items, such as procedures and steps.
Styles of the section titles in a user manual and help files.
• What does not need to be fixed:
Ending styles of bulleted items.
Placeholders, such as &ProductNameShort; and &ProductNameLong;
Punctuation.
http://doras.dcu.ie/16062/1/SAKURA_final_revised.pdf
Porto (14/05/2013) SMT and PE 58 / 67

59. Monolingual PE Guidelines - ACCEPT project
• Try and edit the text by making it more fluent and clearer based on
how you interpret its meaning. For example, try to rectify word order
and spelling when they are inappropriate to the extent that the text
has become impossible or difficult to comprehend
• If words, phrases, or punctuation in the text are completely
acceptable, try and use them (unmodified) rather than substituting
them with something new and different.
http://www.accept.unige.ch/Products/D6.2.2_Seminar_
Material_on_Post-Editing_Edition_2.pdf
Porto (14/05/2013) SMT and PE 59 / 67

60. PE Technology (tools)
A myriad of tools appeared in the marked in the last couple of years:
• TM tools that now integrate MT, e.g. SDL TRADOS, WordFast
• MT tools (with or without TM): e.g. Systran, Google Translate
Toolkit, Caitra
• PE tools that can use different MT systems: MemoQ, MemSource,
SmartMate, Coach, ...
More and more functionalities: spell/grammar checkers, multiple MT,
MT+TM, interactive MT, terminology, word/phrase alignment, etc.
Limited (if any) logging of translator’s activities for productivity analysis –
not enough for research.
Porto (14/05/2013) SMT and PE 60 / 67

61. PE Technology (tools)
Research-oriented tools:
• Translog (not open source)
• OmegaT (TM only)
• Appraise (MT eval through ranking, no detailed statistics for PE)
• PET: http://pers-www.wlv.ac.uk/~in1676/pet/index.html
• Coming soon: open source tool promised as part of EU projects
CASMACAT & MATECAT
Porto (14/05/2013) SMT and PE 61 / 67

62. What can we get from PET?
Comparison between HT and multiple MT systems [dSAS11]
• PET to compare 3 MT systems vs translation from scratch
• Sitcom and movie subtitles:
Translating from scratch can be 73% slower than post-editing
SMT systems (Google and Moses) performed the best
How often a system produced an output that was faster to PE than other systems:
System Google Moses Systran Trados
Google - 139 161 187
Moses 69 - 122 164
Systran 69 106 - 145
Trados 48 67 89 -
How often post-editing a system output was faster than translating:
System Faster than human translation
Google 94%
Moses 86.8%
Systran 81.20%
Trados 72.40%
Porto (14/05/2013) SMT and PE 62 / 67

63. What can we get from PET?
Evaluation of an MT system for translation of subtitles (space constraints)
[AdSS12]
• Add compression constraints to an MT system to generate length
compliant subtitles
• PET to guide post-editing according to length and time requirements
for every unit
Show space limitation (attribute)
Change colour of translation if too long
Offer shorter paraphrases
• PET to evaluate edit distance and length
System
Dexter How I Met.. Terra Nova
TER ↓ LENGTH TER ↓ LENGTH TER ↓ LENGTH
Mosest 30.3 116.0 20.0 108.5 33.8 120.2
Google 63.6 156.5 52.8 144.3 63.1 152.1
MosesLP2 29.5 115.5 21.0 109.1 33.4 119.3
MosesLP1 28.3 115.8 20.7 110.0 34.8 119.8
Porto (14/05/2013) SMT and PE 63 / 67

64. Hands-on activity with PET
• en-pt-hiding-and-assessing.pec – bigbang-pe-2.pej
• en-pt-hiding-and-assessing.pec – bigbang-ht-2.pej
• fapesp.pec – fapesp-annotated.pej
• fapesp-mono.pec – fapesp-annotated.pej
• subs.pec – greys.08x06.v1-v1.en-br.s2.1.pej
Advanced use:
• Configuration file: .pec
• Input/output XML files
• How to build a PEJ file from raw files: pej.bat
• How to interpret the results: scripts/readper.pl
Porto (14/05/2013) SMT and PE 64 / 67

65. For more on the research-side
Workshop on Post-Editing Technology and Practice:
• AMTA 2012: https://sites.google.com/site/wptp2012/
• MT Summit 2013 (deadline for papers 28/05/2013):
https://sites.google.com/site/mts2013wptp/
Porto (14/05/2013) SMT and PE 65 / 67

66. References I
Wilker Aziz, Sheila Castilho Monteiro de Sousa, and Lucia Specia.
Cross-lingual sentence compression for subtitles.
In The 16th Annual Conference of the European Association for Machine
Translation, EAMT ’12, pages 103–110, Trento, Italy, May 2012.
Autodesk.
Translation and Post-Editing Productivity.
In http: // translate. autodesk. com/ productivity. html , 2011.
Sheila C. M. de Sousa, Wilker Aziz, and Lucia Specia.
Assessing the post-editing effort for automatic and semi-automatic
translations of DVD subtitles.
In Conference Recent Advances in Natural Language Processing 2011, pages
97–103, Hissar, Bulgaria, 2011.
Porto (14/05/2013) SMT and PE 66 / 67

67. References II
Maarit Koponen, Wilker Aziz, Luciana Ramos, and Lucia Specia.
Post-editing time as a measure of cognitive effort .
In AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP
2012), pages 11–20, San Diego, USA, October 2012. Association for
Machine Translation in the Americas (AMTA).
Maarit Koponen.
Comparing human perceptions of post-editing effort with post-editing
operations.
In Proceedings of the Seventh Workshop on Statistical Machine Translation,
pages 181–190, Montr´eal, Canada, 2012.
R. Skadins, M. Purins, I. Skadina, and A. Vasiljevs.
An evaluation of SMT in localization to under-resourced inflected languages.
In Proceedings of the 15th Conference of the European Association for
Machine Translation, Leuven, 2011.
Porto (14/05/2013) SMT and PE 67 / 67