Elamur Research⌕
☰
home / papers / 2010 / MultiUN: A Multilingual Corpus from United Nation Documents

MultiUN: A Multilingual Corpus from United Nation Documents

Summary

Statistical MT and other cross-lingual tasks need large, clean parallel corpora. Before this work, such multi-way resources existed mostly for European languages. The authors built MultiUN from the United Nations Official Document System (ODS). It covers all six UN official languages (Arabic, Chinese, English, French, Russian, Spanish) plus a small German portion.

To build it, they crawled Word documents published from 2000 onward and converted them to XML, dropping images, tables and style markup. They then filtered out files with wrong language labels and split the text into sentences. Sentences were aligned with hunalign for all 21 language pairs. Parallel documents were matched through their shared UN document symbols.

The release contains about 463K documents and roughly 300 million words per language. It also includes a reserved, time-based test portion.

To show the corpus is usable, the authors trained Moses systems for Spanish↔Chinese and French↔Chinese. These reached test BLEU scores between about 30 and 39.

The corpus matters because it links European resources to Arabic and Chinese, filling a gap that Europarl and JRC-Acquis leave.

Key points

  • MultiUN version 1 has 463,406 XML documents and 80,931,645 sentences. Five of the languages average about 326 million tokens each. Chinese is counted in characters because segmentation standards differ.
  • Training data covers January 2000 to September 2009. The test material, kept separate from training, consists of documents published between October 2009 and March 2010. Only about 1% of documents are in German.
  • The processing tools were:
    • mguesser (a C implementation of TextCat) for language identification, with custom models built from about 500Kb of verified files;
    • NLTK for sentence splitting, and regular expressions for Chinese;
    • hunalign in two-pass mode for sentence alignment.
  • For the MT experiments, the authors used Moses with SRILM 5-gram language models and MERT tuning on 1,000 sentences, testing on 2,000 sentences. Test BLEU scores were:
    • Spanish→Chinese: 31.35
    • Chinese→Spanish: 39.08
    • French→Chinese: 29.94
    • Chinese→French: 34.66
  • The corpus was released through the EuroMatrixPlus project website. The documents are public domain under UN administrative instruction ST/AI/189/Add.9/Rev.2.

Interesting ideas

  • I1. Shared identifiers for document alignment. Every language version of a UN document carries the same official symbol, so matching parallel documents needs no content comparison. Any multilingual archive with stable shared IDs can be aligned this cheaply.
  • I2. A rolling test set. The newest six months of documents are held out for evaluation, then moved into training at the next update. This keeps benchmarks current and the training set growing, which suits any continuously published source.
  • I3. Discarding whole problem segments. Annexes were often copied rather than translated, so the authors simply dropped documents containing them. When one structural element causes most of the noise, removing it wholesale can beat fine-grained filtering if quality matters more than volume.
  • I4. Not trusting source metadata. The language labels in ODS were sometimes wrong, so the authors checked every file with a small, hand-verified n-gram language-ID model. This is a cheap safeguard for any scraped multilingual collection.
  • I5. Alignment without a bilingual dictionary. hunalign's two-pass mode first aligns by sentence length, then learns a dictionary from that alignment and realigns. This works for language pairs that lack lexical resources, which is relevant for lower-resource pairs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

114 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 101 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{eisele2010multiun,
  title = {{MultiUN: A Multilingual Corpus from United Nation Documents}},
  author = {Eisele, Andreas and Chen, Yu},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/5co5op3ipz42},
  url = {https://doi.org/10.63317/5co5op3ipz42}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexander Barkov. 2008. mguesser. http://www.mnogosearch.org/guesser/. ↗
  2. Steven Bird and Edward Loper. 2004. NLTK: the natural language toolkit. In ACL 2004 on Interactive poster and demonstration sessions, page 31, Morristown, NJ, USA. Association for Computational Linguistics. find
  3. William B. Cavnar and John M. Trenkle. 1994. N-gram-based text categorization. In SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, pages 161–175. find
  4. Yu Chen, Andreas Eisele, and Martin Kay. 2008. Improving Statistical Machine Translation Efficiency by Triangulation. In the 6th International Conference on Language Resources and Evaluation (LREC ’08), May. find
  5. Yu Chen, Martin Kay, and Andreas Eisele. 2009. Inter-Arabic 65156 11050313 237412090 find
  6. Russian 77061 13852535 278606813 Chinese 65022 10839473 756108566 German 3763 232225 5848668 ar 63257 63193 63241 ru 74053 73973 64230 63194 zh 62815 62738 62707 63031 62842 de 3643 3632 3632 3677 3635 3886 ar 8206568 8841529 8794485 ru 6166942 8244246 6873585 9045096 zh 8829060 8727150 8788366 7581… find
  7. Trevor Cohn and Mirella Lapata. 2007. Machine Translation by Triangulation: Making Effective Use of Multi-Parallel Corpora. In the 45th Annual Meeting of the Association for Computational Linguistics, Prague, Czech, June. find
  8. Andreas Eisele. 2006. Parallel Corpora and Phrase-Based Statistical Machine Translation for New Language Pairs via Multiple Intermediaries. In LREC 2006, Genoa, Italy, May. find
  9. David Graff. 1994. UN Parallel Text (Complete). Linguistic Data Consortium, Philadelphia. find
  10. 国民议会第二次常会于5月4日至22日举行。 La deuxième session spéciale de l’Assemblée nationale a eu lieu du 4 au 22 mai. L’Assemblèe nationale à la deuxième réunion ordinaire du 4 au 22 mai . La deuxième session ordinaire de l’Assemblée nationale le 4 Mai 22 a été tenue. Source: Reference: In-house: Online:… find
  11. Natalia Klyueva and Ondřej Bojar. 2008. UMC 0.1: Czech-Russian-English Multilingual Corpus. In International Conference Corpus Linguistics, pages 188– 195, October. find
  12. Philipp Koehn, Hieu Hoang, Alexandra Birch Mayne, Christopher Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbs. 2007. Moses: Open source toolkit for statistical machine translati… find
  13. Philipp Koehn, Alexandra Birch, and Ralf Steinberger. 2009. 462 Machine Translation Systems for Europe. In Machine Translation Summit XII, pages 65–72, August. find
  14. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In MT Summit 2005. find
  15. Shankar Kumar, Franz Josef Och, and Wolfgang Macherey. 2007. Improving word alignment with bridge languages. In the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 42–50, Prague, Czech. find
  16. Franz Josef Och and Hermann Ney. 2001. Statistical multi-source translation. In MT Summit VIII, Santiago de Compostela, Spain. find
  17. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In ACL ’03: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, pages 160–167, Morristown, NJ, USA. Association for Computational Linguistics. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: a method for automatic evaluation of machine translation. In the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Morristown, NJ, USA. Association for Computational Linguistics. find our card
  19. Alexandre Rafalovitch and Robert Dale. 2009. United nations general assembly resolutions: A six-language parallel corpus. In Proceedings of the MT Summit XII, pages 292–299. International Association of Machine Translation, August. find
  20. Michel Simard. 1999. Text-translation alignment: Three languages are better than two. In EMNLP/VLC-99, College Park, MD, June. find
  21. Benjamin Snyder and Regina Barzilay. 2008. Cross-lingual propagation for morphological analysis. In AAAI’08: Proceedings of the 23rd national conference on Artificial intelligence, pages 848–854. AAAI Press. find
  22. Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaz Erjavec, Dan Tufis, and Daniel Varga. 2006. The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages. In LREC 2006, Sep. find our card
  23. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In the 7th International Conference on Spoken Language Processing (ICSLP) 2002, Denver, Colorado. find
  24. Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter. In Fourth SIGHAN Workshop on Chinese Language Processing. find
  25. United Nations Secretariat. 1987. Regulation for the control and limitation of documentation. Administrative Instruction (ST/AI/189/Add.9/Rev.2), 17 September. find
  26. Dániel Varga, László Németh, Péter Halácsy, András Kornai, Viktor Trón, and Viktor Nagy. 2005. Parallel corpora for medium density languages. In RANLP, pages 560–596, Borovets, Bulgaria. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/eisele-2010-multiun-multilingual-corpus-united-nation/