Findings of the WMT 2018 Shared Task on Automatic Post-Editing
Summary
This report describes the fourth edition of the WMT shared task on Automatic Post-Editing (APE). In APE, a system repairs the output of an inaccessible "black-box" MT engine by learning from professional corrections. The 2018 round used English-German data from the Information Technology domain and split into two subtasks. One subtask covered output from a phrase-based system (PBSMT), and the other covered output from a neural system (NMT), which was included for the first time. Participants received small in-domain sets of (source, translation, post-edit) triplets plus large synthetic corpora.
Five teams entered, and every submission was a neural model built on the Transformer, sometimes with multi-source inputs. On PBSMT output, the best systems substantially improved on the do-nothing baseline, exceeding the 2017 gains. On NMT output, improvements were marginal. The organizers attribute this to two factors: the neural translations were already of higher quality, and less in-domain training data was available. Manual direct assessment broadly agreed with the automatic metrics. The results suggest that APE becomes harder as base MT quality improves.
Key points
- Participation: 5 teams took part, with the PBSMT subtask receiving 11 runs and the NMT subtask receiving 10.
- Data:
- PBSMT training used 28,000 triplets reused from the 2016–2017 rounds, with a test set of 2,000 instances.
- NMT had 13,442 training, 1,000 development and 1,023 test triplets.
- Extra resources were the 4.5M-triplet "Artificial" corpus and the English-German eSCAPE corpus, which has 7.25M PBSMT and 7.25M NMT instances.
- Baseline quality (TER/BLEU): 24.24/62.99 for the PBSMT test set versus 16.84/74.73 for NMT. About 25.2% of NMT test items needed no correction, so any unnecessary edit was penalized.
- Top automatic gains: −6.24 TER and +9.53 BLEU on PBSMT, compared with only −0.38 TER and +0.8 BLEU on NMT.
- Human evaluation (direct assessment, DA):
- MS UEdin ranked first in both subtasks, scoring 93.27% on PBSMT and 91.11% on NMT. Human post-edits scored 95.87% and 96.13%.
- On NMT, no system differed significantly from the baseline.
Cited by
- Findings of the 2021 Conference on Machine Translation (WMT21)
- Findings of the 2018 Conference on Machine Translation (WMT18)
- Iterative Translation Refinement with Large Language Models
- Diagnose, Then Repair: A Two-Stage MQM-Guided Post-Editing Framework for Domain-Specific Machine Translation
Cite
@inproceedings{chatterjee2018findings,
title = {{Findings of the WMT 2018 Shared Task on Automatic Post-Editing}},
author = {Chatterjee, Rajen and Negri, Matteo and Rubino, Raphael and Turchi, Marco},
booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
year = {2018},
doi = {10.18653/v1/w18-6452},
url = {https://doi.org/10.18653/v1/w18-6452}
}References
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
- Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus Phrase-Based Machine Translation Quality: a Case Study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 257–267, Austin, Texas, November. Association for Comp… find our card
- Besides the amount of training data (the training corpora for the two subtasks have different size), the task difficulty is proportional to the quality of the initial translations. In line with previous years, learning from (and testing on) lower quality data leaves more room for improvement. Ondře… find our card
This page in Markdown → https://md.research.elamur.ai/free/papers/chatterjee-2018-findings-wmt-2018-shared-task/