Elamur Research⌕
☰
home / papers / 2022 / MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset

MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset

Summary

MLQE-PE is a public dataset for training and testing machine translation Quality Estimation (QE), which predicts how good a translation is without a reference. It also supports Automatic Post-Editing (APE). The authors list four gaps in earlier WMT QE data: the MT systems were closed, each set had only one kind of quality label, most pairs were high-resource, and the domains were narrow enough that most outputs were good.

The new resource covers 11 language pairs, mostly built from Wikipedia sentences and translated with fairseq Transformer models. These models are released along with their sentence scores and word log-probabilities. The same segments carry three kinds of labels:

  • sentence-level direct assessments (DA) from several annotators,
  • post-edits with HTER,
  • word-level OK/BAD tags.

Each segment also comes with the title of its source article. The data spans high-, medium- and low-resource pairs and includes zero-shot test sets. Baseline QE systems built with OpenKiwi are reported. The dataset was the basis for the WMT 2020 and 2021 QE shared tasks, which makes it a common benchmark.

Key points

  • Language pairs: En-De, En-Zh, Ru-En, Ro-En, Et-En, Ne-En and Si-En each have about 7K train, 1K dev, 1K test20 and 1K test21 sentences. Ps-En, Km-En, En-Ja and En-Cs are test-only zero-shot pairs with 2K sentences each, half of which is held back as a blind WMT QE 2022 test set.
  • Source selection: sentences follow the FLORES sampling procedure, are 50–150 characters long, and were filtered against WikiMatrix so they do not overlap common NMT training data.
  • En-De and En-Zh sampling: these pairs were sampled with weights based on perplexity against the NMT training data, so that both good and poor translations appear.
  • Ru-En data: this pair draws on Reddit (politics and religion subreddits) and Russian proverbs from WikiQuote. Proverbs scored far lower on average DA (41.1) than Reddit sources (about 74–80).
  • Error density: in Ne-En, Si-En and Km-En, roughly 96–100% of target sentences contain at least one BAD tag. For Et-En, a 1K subset also has two independent professional reference translations.

Interesting ideas

  • I1. Release the generating MT model with the data. This allows glass-box QE based on model confidence or uncertainty, rather than treating the system as a black box. It is worth copying for any MT evaluation set we build.
  • I2. Put several label types on the same segments. Combining DA, post-edits and word tags lets researchers study how perceived quality relates to correction effort. Keeping individual annotator scores also supports modelling annotator disagreement.
  • I3. Sample by perplexity to get a realistic quality mix. For strong language pairs, weighting the sample toward sentences unlike the training data avoids a test set full of easy, near-perfect translations.
  • I4. Add deliberately hard text as stress tests. Proverbs (short, with odd grammar) and colloquial Reddit posts expose MT weaknesses. A Ukrainian dataset could use idioms or informal social-media text in the same way.
  • I5. Count gap errors without enlarging the denominator. When BAD tags on insertion/deletion gaps are counted, keeping the token total unchanged avoids understating word-level error rates. This is a useful convention for any token-level error statistic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

62 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 50 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{fomicheva2022mlqe,
  title = {{MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset}},
  author = {Fomicheva, Marina and Sun, Shuo and Fonseca, Erick and Zerva, Chrysoula and Blain, Frédéric and Chaudhary, Vishrav and Guzmán, Francisco and Lopatina, Nina and Specia, Lucia and Martins, André F. T.},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/2av2mm9gpjb9},
  url = {https://doi.org/10.63317/2av2mm9gpjb9}
}

References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akhbardeh, F., Arkhangorodsky, A., Biesialska, M., Bojar, O., Chatterjee, R., Chaudhary, V., Costa-jussa, M. R., España-Bonet, C., Fan, A., Federmann, C., Freitag, M., Graham, Y., Grundkiewicz, R., Haddow, B., Harter, L., Heafield, K., Homan, C., Huck, M., Amponsah-Kaakyire, K., Kasai, J., Khashabi, D., Knight, K., Kocmi, T., Koehn, P., Lourie, N., Monz, C., Morishita, M., Nagata, M., Nagesh, A., Nakazawa, T., Negri, M., Pal, S., Tapo, A. A., Turchi, M., Vydrin, V., and Zampieri, M. (2021). Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1–88, Online, November. Association for Computational Linguistics. find our card
  2. Blatz, J., Fitzgerald, E., Foster, G., Gandrabur, S., Goutte, C., Kulesza, A., Sanchis, A., and Ueffing, N. (2004). Confidence Estimation for Machine Translation. In Proc. of the International Conference on Computational Linguistics, page 315. find
  3. Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. (2014). Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58, Baltimore, Maryland, USA, June. Association for Computational Linguistics. find our card
  4. Chatterjee, R., Federmann, C., Negri, M., and Turchi, M. (2019). Findings of the wmt 2019 shared task on automatic post-editing. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 11–28. find our card
  5. Chatterjee, R., Freitag, M., Negri, M., and Turchi, M. (2020). Findings of the WMT 2020 shared task on automatic post-editing. In Proceedings of the Fifth Conference on Machine Translation, pages 646–659, Online, November. Association for Computational Linguistics. find
  6. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online, July. Association for Computational Linguistics. find
  7. Esplà, M., Forcada, M., Ramı́rez-Sánchez, G., and Hoang, H. (2019). ParaCrawl: Web-scale parallel corpora for the languages of the EU. In Proceedings of Machine Translation Summit XVII Volume 2: Translator, Project and User Tracks, pages 118–119, Dublin, Ireland, 19–23 August. European Association for Machine Translation. find our card
  8. Fomicheva, M., Specia, L., and Guzmán, F. (2020a). Multi-hypothesis machine translation evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1218– 1232. find
  9. Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzmán, F., Fishel, M., Aletras, N., Chaudhary, V., and Specia, L. (2020b). Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computation Linguistics. ↗ our card
  10. Fomicheva, M., Lertvittayakumjorn, P., Zhao, W., Eger, S., and Gao, Y. (2021). The eval4nlp shared task on explainable quality estimation: Overview and results. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 165–178. find
  11. Fonseca, E., Yankovskaya, L., Martins, A. F. T., Fishel, M., and Federmann, C. (2019). Findings of the WMT 2019 shared tasks on quality estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 1–10, Florence, Italy, August. Association for Computational Linguistics. find our card
  12. Graham, Y., Baldwin, T., Moffat, A., and Zobel, J. (2013). Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41. find
  13. Guzmán, F., Chen, P.-J., Ott, M., Pino, J., Lample, G., Koehn, P., Chaudhary, V., and Ranzato, M. (2019). The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6097–6110, Hong Kong, China, November. Association for Computational Linguistics. find our card
  14. Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. find
  15. Kepler, F., Trénous, J., Treviso, M., Vera, M., and Martins, A. F. T. (2019). OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117–122, Florence, Italy, July. Association for Computational Linguistics. find our card
  16. Kim, H., Lee, J.-H., and Na, S.-H. (2017). Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation. In Conference on Machine Translation (WMT). find our card
  17. Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pages 79–86. find
  18. Matthews, B. W. (1975). Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. ↗
  19. Ng, N., Yee, K., Baevski, A., Ott, M., Auli, M., and Edunov, S. (2019). Facebook fair wmt19 news translation task submission. In Proc. of WMT, pages 1–4. find
  20. Ott, M., Edunov, S., Grangier, D., and Auli, M. (2018). Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1–9. find our card
  21. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53. ↗ our card
  22. Sabet, M. J., Dufter, P., Yvon, F., and Schütze, H. (2020). Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2020. ↗
  23. Schwenk, H., Chaudhary, V., Sun, S., Gong, H., and Guzmán, F. (2019). WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia. arXiv preprint arXiv:1907.05791. ↗ our card
  24. Specia, L., Blain, F., Logacheva, V., Astudillo, R., and Martins, A. F. (2018a). Findings of the wmt 2018 shared task on quality estimation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 689–709. find our card
  25. Specia, L., Scarton, C., and Paetzold, G. H. (2018b). Quality Estimation for Machine Translation. Synthesis Lectures on Human Language Technologies, 11(1):1–162. ↗
  26. Specia, L., Blain, F., Fomicheva, M., Fonseca, E., Chaudhary, V., Guzmán, F., and Martins, A. F. T. (2020). Findings of the WMT 2020 shared task on quality estimation. In Proceedings of the Fifth Conference on Machine Translation, pages 743–764, Online, November. Association for Computational Linguistics. find our card
  27. Specia, L., Blain, F., Fomicheva, M., Zerva, C., Li, Z., Chaudhary, V., and Martins, A. F. T. (2021). Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 684–725, Online, November. Association for Computational Linguistics. find our card
  28. Tang, Y., Tran, C., Li, X., Chen, P., Goyal, N., Chaudhary, V., Gu, J., and Fan, A. (2020). Multilingual translation with extensible multilingual pretraining and finetuning. ArXiv, abs/2008.00401. ↗
  29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems, pages 5998– 6008. find
  30. Wang, K., Shi, Y., Wang, J., Zhang, Y., Zhao, Y., and Zheng, X. (2021). Beyond glass-box features: Uncertainty quantification enhanced quality estimation for neural machine translation. arXiv preprint arXiv:2109.07141. ↗
  31. Zerva, C., van Stigt, D., Rei, R., Farinha, A. C., Ramos, P., de Souza, J. G., Glushkova, T., Vera, M., Kepler, F., and Martins, A. F. (2021). Ist-unbabel 2021 submission for the quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 961–972. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/