CUNI Transformer Neural MT System for WMT18

Summary

This paper describes CUNI Transformer, Charles University's system for the English→Czech and Czech→English news translation task at WMT2018. The author used only the constrained WMT data: about 58M authentic sentence pairs, mostly from CzEng 1.7, plus Czech and English NewsCrawl monolingual text of similar size. The model is a big Transformer in Tensor2Tensor, trained with the Adafactor optimizer and checkpoint averaging.

Five lightweight techniques improved it:

cleaning the synthetic data,
an iterated backtranslation scheme in which authentic and synthetic corpora are simply concatenated rather than shuffled,
picking separate checkpoints for Czech-origin and foreign-origin documents,
coreference-driven insertion of dropped Czech female pronouns,
regex-based output fixes.

The system ranked first in both directions at WMT2018 and was significantly better than the runner-up, UEdin NMT, on both BLEU and human direct assessment. A small manual check suggested its output can match or exceed the human references. This makes single-reference automatic metrics questionable at this quality level. All five techniques are architecture-agnostic.

Key points

Data filtering (~3% of synthetic sentences removed):
- Czech-side sentences without any accented character were dropped, which caught misplaced English text.
- Sentences over 500 characters were removed.
- Sentences with words repeated more than twice were filtered with a regex.
Iterated "concat" backtranslation: Nematus2016 → English→Czech Transformer → Czech→English Transformer → final English→Czech model. Each training run took eight days on eight GTX 1080 Ti GPUs. Better synthetic data added +0.83 BLEU on newstest2017.
Origin-based checkpoint selection: checkpoints were chosen on the Czech-origin and non-Czech-origin parts of newstest2013 (774k and 788k steps). This added +0.21 BLEU, reaching 26.81 cased BLEU on newstest2017, versus 22.80 for the WMT17 winner Nematus.
Coreference pre-processing: Treex with its coreference resolver inserted "ona" (she) where a human antecedent was female. This touched about 1% of sentences and left BLEU unchanged, though the author judged most of those translations improved.
WMT2018 newstest2018 results:

Direction Cased BLEU refDA UEdin NMT (BLEU / refDA)

English→Czech 26.01 67.2 23.42 / 60.6

Czech→English 33.91 71.8 33.06 / 67.9

The gap over UEdin NMT is significant at p < 0.05 in both directions.

Direction	Cased BLEU	refDA	UEdin NMT (BLEU / refDA)
English→Czech	26.01	67.2	23.42 / 60.6
Czech→English	33.91	71.8	33.06 / 67.9

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Coreference resolution · Machine translation

Cited by

27 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 22 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6424

@inproceedings{popel2018cuni,
  title = {{CUNI Transformer Neural MT System for WMT18}},
  author = {Popel, Martin},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Shared Task Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6424},
  url = {https://doi.org/10.18653/v1/w18-6424}
}

References

8 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of EMNLP, volume 4, pages 388–395. find
Shaohui Kuang, Deyi Xiong, Weihua Luo, and Guodong Zhou. 2017. Cache-based Document-level Neural Machine Translation. CoRR, arXiv/1711.11221. find
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational… find
Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 Metrics Shared Task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria. Association for Computational Linguistics. find
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. CoRR, arXiv/1804.04235. find
Michal Novák. 2017. Coreference resolution system not only for czech. In Proceedings of the 17th conference ITAT 2017: Slovenskočeský NLP workshop (SloNLP 2017), pages 193–200, Praha, Czechia. CreateSpace Independent Publishing Platform. find
Roman Sudarikov, Martin Popel, Ondřej Bojar, Aljoscha Burchardt, and Ondřej Klejch. 2016. Using MT-ComparEval. In Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem, pages 76–82, Portorož, Slovenia. LREC. find
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 6000–6010. Curran Associates, Inc. find

This page in Markdown → https://md.research.elamur.ai/free/papers/popel-2018-cuni-transformer-neural-mt-system/

📄 CUNI Transformer Neural MT System for WMT18