Beyond BLEU: Professional Error Annotation in MT Quality Estimation and Improvement
•Als PPTX, PDF herunterladen•
2 gefällt mir•1,738 views
The document discusses the limitations of automated metrics like BLEU for evaluating machine translation quality and proposing alternatives like professional human annotation of errors in machine translation using frameworks like MQM/DQF. It presents error profiles of machine translation systems based on such annotations, which can provide more useful information than automated scores to understand issues and improve translation quality. The document advocates moving beyond fully automated metrics to involve human analysis of errors and source language phenomena to build test suites that help determine relationships between source barriers and target errors to enhance machine translation systems.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Beyond BLEU: Professional Error Annotation in MT Quality Estimation and Improvement
Ähnlich wie Beyond BLEU: Professional Error Annotation in MT Quality Estimation and Improvement (20)
Mehr von TAUS - The Language Data Network
Mehr von TAUS - The Language Data Network (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Beyond BLEU: Professional Error Annotation in MT Quality Estimation and Improvement
- 1. Kim Harris with Aljoscha Burchardt (DFKI), Hans Uszkoreit (DFKI), Arle Lommel (CSA) BEYOND AUTOMATED QUALITY SCORES From BLEU to professional error annotation in MT quality estimation and improvement
- 2. Kim Harris • TAUS Roundtable Vienna 2016 • The closer a machine translation is to a professional human translation, the better it is • Relatively high correlation with human judgements • One of the most popular automated and inexpensive metrics. • Automated quality scores based on comparisons with sets of HT references • Can be useful for certain estimation tasks but not for improvement • No ability to assess why scores improve or worsen • Focus on the score and not the actual quality BLEU: Status Quo 2
- 3. Kim Harris • TAUS Roundtable Vienna 2016 • MQM/DQF error annotation for HT and MT • Analysis of quality based on real issues • Ranking/estimation properties • Use results to improve output Error Annotation for MT Improvement 3
- 4. Kim Harris • TAUS Roundtable Vienna 2016 Annotation: Humans in the HQMT loop 4
- 5. Kim Harris • TAUS Roundtable Vienna 2016 Error profiles based on MQM annotation By languages By system types 5
- 6. Kim Harris • TAUS Roundtable Vienna 2016 6 Error profiles
- 7. Kim Harris • TAUS Roundtable Vienna 2016 Error and source barrier analysis • Moving away from completely automatic • Analyse MQM errors, linguistic phenomena in target MT • Compare to source phenomena • Test suite analysis • Basis for improved quality translation in MT thanks to categorization and markup of translation barriers in source language • Mapping (almost) all linguistic phenomena for one language • determine possible relationships between phenomena in the source and errors in the target • can be used to test different MT systems and domains New paradigm in HQMT 7
- 8. Kim Harris • TAUS Roundtable Vienna 2016 Enter: The Test Suite 8
- 9. Kim Harris • TAUS Roundtable Vienna 2016 Structure of Barrier Categories 9
- 10. Kim Harris • TAUS Roundtable Vienna 2016 Beyond BLEU 10
- 11. Kim Harris • TAUS Roundtable Vienna 2016 The Bigger Vision 11
- 12. Quality Translation 21 (QT21) has received funding from the EU’s Horizon 2020 research and innovation programme under grant no. 645452. META-QT has received funding from the EU’s Horizon 2020 research and innovation programme through the contract CRACKER (grant agreement no.: 645357). Formerly co-funded by FP7 and ICT PSP through the contracts T4ME (grant agreement no.: 249119), CESAR (grant agreement no.: 271022), METANET4U (grant agreement no.: 270893) and META-NORD (grant agreement no.: 270899). Thank you!