Elamur Research⌕
☰
home / papers / 2012 / DGT-TM: A freely available Translation Memory in 22 languages

DGT-TM: A freely available Translation Memory in 22 languages

Summary

Highly multilingual parallel data is scarce, yet it underpins statistical machine translation, annotation projection and cross-lingual text mining. This paper documents DGT-TM Release 2011, a translation memory published jointly by the European Commission's Directorate General for Translation (DGT) and Joint Research Centre (JRC). It covers 22 official EU languages, which gives 231 language pairs. The source material is legislative documents from the L-series of the EU Official Journal, issued 2004–2010. Professional translators produced and revised these texts under strict terminology control. DGT's in-house aligner split them into Translation Units (TUs). The aligner relies on document structure, numbering and other non-linguistic anchors, and it falls back on sentence-length statistics when no anchors are present. Every language was aligned against English, and the bundled tool derives the non-English pairs from those links. The release is about twice as large as the 2007 edition and does not repeat any of its documents. It is distributed as TMX files inside zip packages. The resource matters because it supplies uniform, high-quality parallel data for many language pairs that are rarely covered elsewhere.

Key points

  • Coverage: 22 EU languages and 231 language pairs, drawn from Official Journal L-series (legislation) documents published 2004–2010. Documents already in the 2007 release were excluded.
  • The release is roughly double the size of DGT-TM 2007. Annual releases are planned, along with processing of the C-series and a full-text (document-order) version.
  • About 16–17% of TUs are sentence fragments split at colons or semicolons, which is typical of legal texts. The authors nonetheless treat TUs as equivalent to sentences.
  • English served as the hub for alignment: each of the other 21 languages was aligned to it. No exact alignment accuracy figures exist. Quality was judged in production, where translators reported errors that were then used to refine the aligner.
  • Distribution: 25 zip packages of at most 100 MB each. TMX files are named by EUR-Lex identifier. An extraction program reads the zips directly, but users need all packages even for a single language pair. In 2008, English was the original drafting language for 72% of source documents, with French accounting for 11%.

Interesting ideas

  • I1. Pivot-based alignment: Aligning every language only to one hub language and deriving the other pairs from those links cuts the work from hundreds of pairwise alignments to about twenty. This could suit any multilingual collection with a dominant source language.
  • I2. Structure-first alignment: The aligner uses document numbering to define zones, then strengthens matches with numbers, images and other non-textual anchors. Length statistics are used only as a last resort. The approach fits well with structured documents such as laws, standards, customs regulations or technical manuals.
  • I3. Users as evaluators: Instead of building a formal test set, the team showed alignments to working translators and fed their error reports back into the algorithm. This is a cheap, continuous quality loop for resources that are already used in production.
  • I4. Incremental, non-overlapping releases: Each yearly release excludes documents published in earlier versions. This lets users accumulate data without duplicates and keeps train/test separation manageable.
  • I5. Reporting fragment indicators: Counting colons and semicolons signals how many units are sentence parts rather than full sentences. This simple statistic is worth including when documenting any segmented corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

67 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{steinberger2012dgt,
  title = {{DGT-TM: A freely available Translation Memory in 22 languages}},
  author = {Steinberger, Ralf and Eisele, Andreas and Klocek, Szymon and Pilos, Spyridon and Schlüter, Patrick},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/2tewbj4q8j4g},
  url = {https://doi.org/10.63317/2tewbj4q8j4g}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. DGT-TM (2007). http://langtech.jrc.ec.europa.eu/DGT-TM.html. ↗
  2. EC&DGT (2008). European Commission & Directorate General for Translation – Translation Tools and Workflow. Office for Official Publications, Brussels, Belgium. find
  3. Ehrmann Maud, Marco Turchi & Ralf Steinberger (2011). Building a Multilingual Named Entity-Annotated Corpus. Proceedings of the 8 th International Conference Recent Advances in Natural Language Processing (RANLP'2011). Hissar, Bulgaria, 12-14.9.2011. find
  4. Koehn Philipp (2005). EuroParl: A Parallel Corpus for Statistical Machine Translation. Proceedings of Machine Translation Summit. find
  5. Koehn Philipp, Alexandra Birch & Ralf Steinberger (2009). 462 Machine Translation Systems for Europe. Proceedings of 12 th Machine Translation Summit. find
  6. Landauer Thomas & Michael Littman (1991). A Statistical Method for Language-Independent Representation of the Topical Content of Text Segments. Proceedings of the 11th International Conference ‘Expert Systems and Their Applications’, vol. 8: pp. 77-85. find
  7. Macklovitch Elliott, Michel Simard & Philippe Langlais (2000). TransSearch: A Free Translation Memory on the World Wide Web. Proceedings of LREC. find
  8. Parasol (2011). Parasol – A parallel corpus of Slavic and other languages. http://parasol.unibe.ch/. ↗
  9. Potthast Martin, Alberto Barrón-Cedeño, Benno Stein, Paolo Rosso (2011). Cross-Language Plagiarism Detection. In: Language Resources and Evaluation. Special Issue on Plagiarism and Authorship Analysis, vol. 45, num. 1, pp.45-62. find
  10. Steinberger Josef, Polina Lenkova, Mijail Kabadjov, Ralf Steinberger & Erik van der Goot (2011). Multilingual Entity-Centered Sentiment Analysis Evaluated by Parallel Corpora. Proceedings of the 8 th International Conference Recent Advances in Natural Language Processing (RANLP'2011). Hissar, Bulgar… find
  11. Steinberger Ralf (2011). A survey of methods to ease the development of highly multilingual Text Mining applications. Language Resources and Evaluation Journal. Online First on 11 October 2011. find
  12. Steinberger Ralf, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaž Erjavec, Dan Tufiş, Dániel Varga (2006). The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages. Proceedings of the 5 th International Conference on Language Resources and Evaluation (LREC'2006), pp. 2142-2147.… find our card
  13. Steinberger Ralf, Bruno Pouliquen, Mijail Kabadjov & Erik van der Goot (2011). JRC-Names: A freely available, highly multilingual named entity resource. Proceedings of the 8 th International Conference Recent Advances in Natural Language Processing (RANLP'2011), pp. 104-110. Hissar, Bulgaria, 12-14… find
  14. Steinberger Ralf, Mohamed Ebrahim & Marco Turchi (2012). JRC EuroVoc Indexer JEX - A freely available multi-label categorisation tool. Proceedings of the 8 th international conference on Language Resources and Evaluation (LREC'2012), Istanbul, 21-27 May 2012. find
  15. Tiedemann Jörg (2009). News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces. In Recent Advances in Natural Language Processing (vol V), pages 237-248, John Benjamins, Amsterdam/Philadelphia. find
  16. Turchi Marco, Josef Steinberger, Mijail Kabadjov & Ralf Steinberger (2010). Using Parallel Corpora for Multilingual (Multi-Document) Summarisation Evaluation. Multilingual and Multimodal Information Access Evaluation. Springer Lecture Notes for Computer Science, LNCS 6360/2010, pp. 52-63. find
  17. Vinokourov Alexei, John Shawe-Taylor, Nello Cristianini (2002). Inferring a semantic representation of text via cross-language correlation analysis. Advances of Neural Information Processing Systems 15. find
  18. Wei Chih-Ping, Christopher C. Yang, Chia-Min Lin (2008). A Latent Semantic Indexing-based approach to multilingual document clustering. Decision Support Systems 45 (2008), 606–620. find
  19. Yarowsky, D., Ngai, G. and Wicentowski, R. (2001) Inducing Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora. In Proceedings of HLT’01, San Diego. find
  20. Table 2. Number of TUs per language pair. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/steinberger-2012-dgt-tm-freely-available-translation/