Elamur Research
☰
home / papers / 2016 / The United Nations Parallel Corpus v1.0

The United Nations Parallel Corpus v1.0

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/L16-1561.pdf

Summary

UN documents have long been used in machine translation and NLP research. Earlier corpora built from them were unofficial, mostly scraped from the Official Document System (ODS), and often incomplete. This paper presents the United Nations Parallel Corpus v1.0, the first such corpus released by the UN itself (via DGACM). Its human-translated documents date from 1990 to 2014 and span all six UN official languages: English, French, Spanish, Russian, Chinese, and Arabic. The corpus is available under a permissive, disclaimer-based license.

Documents are stored in a TEI-based XML format modeled on JRC-Acquis, with metadata such as symbol, job number, date, processing place, and UNBIS-based keywords. Sentence alignment uses a two-step procedure. First, a bootstrap Moses system translates the non-English side into English. Then BLEU-Champ aligns the English output with the English original.

The authors also release a six-way fully aligned subcorpus. They report Moses phrase-based BLEU baselines for all English pairs and for every direction within the subcorpus. The resource supports work on rarely studied pairs such as Arabic-Chinese, pivoting, multi-source translation, and document-level MT.

Key points

  • Scale: 799,276 documents and 1,727,539 aligned document pairs. Depending on the pair, the corpus is roughly 2–4 times larger than the Chen and Eisele (2012) data.
  • Fully aligned subcorpus: 86,307 documents, 11,365,709 lines, and about 335M English tokens, consistently aligned across all six languages.
  • Alignment pipeline:
    • Hunalign aligns a sample of 10,000 document pairs.
    • These are used to train a Moses system with significance pruning.
    • BLEU-Champ (smoothed sentence-level BLEU-2 with the Champollion algorithm) then aligns sentences.
    • For non-English pairs, both sides are first translated into English.
  • Baseline BLEU from and into English: highest for Spanish (en→es 61.35, es→en 60.38) and lowest for Chinese (en→zh 37.68, zh→en 43.68). Systems use MGIZA++, KenLM, an operation sequence model, and Jieba for Chinese segmentation.
  • Production context: the corpus grew out of TAPTA4UN, the UN's production SMT service, which was developed with WIPO and is integrated into the eLUNa CAT tool.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

226 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 204 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ziemski2016united,
  title = {{The United Nations Parallel Corpus v1.0}},
  author = {Ziemski, Michał and Junczys-Dowmunt, Marcin and Pouliquen, Bruno},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  url = {https://aclanthology.org/L16-1561/}
}

References

15 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen, Y. and Eisele, A. (2012). MultiUN v2: UN documents with multilingual alignments. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey, May. European Language Resources Asso… find
  2. Durrani, N., Fraser, A., Schmid, H., Hoang, H., and Koehn, P. (2013). Can Markov models over minimal translation units help phrase-based SMT? In ACL, pages 399–405. The Association for Computer Linguistics. find
  3. Eisele, A. and Chen, Y. (2010). MultiUN: A multilingual corpus from United Nation documents. In Language Resources and Evaluation. find
  4. Gao, Q. and Vogel, S. (2008). Parallel implementations of word alignment tool. In Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 49–57. ACL. Graliński, F., Jassem, K., and Junczys-Dowmunt, M. (2012). PSI-Toolkit: Natural Language Processing Pipeline. Com… find
  5. Heafield, K., Pouzyrevsky, I., Clark, J. H., and Koehn, P. (2013). Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the ACL, pages 690–696. find our card
  6. Johnson, J. H., Martin, J., Forst, G., and Kuhn, R. (2007). Improving translation quality by discarding most of the phrasetable. In Proceedings of EMNLP-CoNLL’07, pages 967–975. find
  7. Junczys-Dowmunt, M. (2012). Phrasal Rank-Encoding: Exploiting phrase redundancy and translational relations for phrase table compression. Prague Bull. Math. Linguistics, 98:63–74. find
  8. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL,… find
  9. ACL. Ma, X. (2006). Champollion: A robust parallel text sentence aligner. In Proceedings of LREC-2006. find
  10. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. find
  11. Pouliquen, B., Mazenc, C., and Iorio, A. (2011). TAPTA: A user-driven translation system for patent documents based on domain-aware statistical machine translation. In Proceedings of the 15th Annual Conference of the EAMT, pages 5–12, Trento, Italy. find
  12. Pouliquen, B., Mazenc, C., Elizalde, C., and Garcia-Verdugo, J. (2012). Statistical machine translation prototype using UN parallel documents. In 16th Annual Conference of the European Association for Machine Translation (EAMT 2012), pages 12–19, Trento, Italy. Pouliquen, B., Elizalde, C., Junczys-D… find
  13. Pouliquen, B., Junczys-Dowmunt, M., Pinero, B., and Ziemski, M. (2015). SMT at the International Maritime Organization: experiences with combining in-house corpora with out-of-domain corpora. In Proceedings of the 18th Annual Conference of the European Association for Machine Translation, pages 202–… find
  14. Rafalovitch, A. and Dale, R. (2009). United Nations general assembly resolutions: A six-language parallel corpus. In MT Summit XII, pages 292–299. International Association of Machine Translation. find
  15. Sennrich, R. and Volk, M. (2011). Iterative, MT-based sentence alignment of parallel texts. 18th Nordic Conference of Computational Linguistics, NODALIDA. Steinberger, R., Pouliquen, B., Widiger, A., Ignat, C., Erjavec, T., and Tufiş, D. (2006). The JRC-Acquis: A multilingual aligned parallel corpu… find