Elamur Research⌕
☰
home / papers / 2009 / Compiling a Massive, Multilingual Dictionary via Probabilistic Inference

Compiling a Massive, Multilingual Dictionary via Probabilistic Inference

Summary

The authors investigate whether the numerous Wiktionaries and bilingual dictionaries available on the web can be automatically combined into a single multilingual resource that would cover far more than any individual source. The main difficulty lies in transitive inference. If word A translates as B, and B as C, then A and C may have different meanings when B is polysemous. It is therefore necessary to estimate the probability that C is indeed a translation of A. From 631 online dictionaries, the authors built a translation graph: its vertices are "word, language" pairs, and its edges denote a shared sense. The graph contains about 10 million words in more than 1000 languages. For inference, they propose the SenseUniformPaths algorithm. It searches for closed translation cycles of any length, rather than only triangles, as the earlier TransGraph system did. To make the method scale, exhaustive cycle enumeration is replaced by graph sampling. The result is the PanDictionary: more than 200 million translation pairs for more than 200 thousand language pairs at a precision of 0.8. The work shows that lexical translation can be obtained even for languages without parallel corpora.

Key points

  • Sources: 631 online dictionaries, including Wiktionaries, which distinguish word senses, and bilingual dictionaries, which do not. All of them were converted to a common XML format and merged into the translation graph.
  • At a precision of 0.9, SenseUniformPaths finds 3.5 times more inferred translations than TransGraph (Etzioni et al., 2007). In addition, the result is stored as a ready-made dictionary, so translations do not have to be computed at each query.
  • Depending on the required precision, PanDictionary is 4.5–24 times larger than the English Wiktionary. Even at a precision of 0.90, it is more than four times larger than this largest multilingual resource.
  • Compared with the sum of all 631 sources, the number of "word–word" translation pairs increases by 73% at a precision of 0.90 and by a factor of 2.7 at a precision of 0.85. Most translations were inferred by combining several dictionaries. The word sense is also indicated for each translation.
  • The resource is useful for translating search queries and tags, cross-lingual image search, and knowledge-based machine translation systems. It covers far more languages than the English Wiktionary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Multilingual & cross-lingual NLP

Cite

Cite as Mausam, Stephen Soderland and Oren Etzioni. 2009. Compiling a Massive, Multilingual Dictionary via Probabilistic Inference. ACL-IJCNLP 2009 (Singapore).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1687878.1687917

@inproceedings{mausam2009compiling,
  title = {{Compiling a Massive, Multilingual Dictionary via Probabilistic Inference}},
  author = {Mausam, Stephen Soderland and Oren Etzioni},
  booktitle = {ACL-IJCNLP 2009 (Singapore)},
  year = {2009},
  doi = {10.3115/1687878.1687917},
  url = {https://doi.org/10.3115/1687878.1687917}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Adar, M. Skinner, and D. Weld. 2009. Information arbitrage in multi-lingual Wikipedia. In Procs. of Web Search and Data Mining(WSDM 2009). find
  2. C. Andrieu, N. De Freitas, A. Doucet, and M. Jordan. 2003. An Introduction to MCMC for Machine Learning. Machine Learning, 50:5–43. find
  3. F. Bond, S. Oepen, M. Siegel, A. Copestake, and D D. Flickinger. 2005. Open source machine translation with DELPH-IN. In Open-Source Machine Translation Workshop at MT Summit X. find
  4. J. Carbonell, S. Klein, D. Miller, M. Steinbaum, T. Grassiany, and J. Frey. 2006. Context-based machine translation. In AMTA. find
  5. A. Copestake, T. Briscoe, P. Vossen, A. Ageno, I. Castellon, F. Ribas, G. Rigau, H. Rodriquez, and A. Samiotou. 1994. Acquisition of lexical translation relations from MRDs. Machine Translation, 3(3–4):183–219. find
  6. H. Dyvik. 2004. Translation as semantic mirrors: from parallel corpus to WordNet. Language and Computers, 49(1):311–326. find
  7. O. Etzioni, K. Reiter, S. Soderland, and M. Sammer. 2007. Lexical translation with application to image search on the Web. In Machine Translation Summit XI. find
  8. M. Franz, S. McCarly, and W. Zhu. 2001. English-Chinese information retrieval at IBM. In Proceedings of TREC 2001. find
  9. P. Fung. 1995. A pattern matching method for finding noun and proper noun translations from noisy parallel corpora. In Proceedings of ACL-1995. find
  10. W. Gale and K.W. Church. 1991. A Program for Aligning Sentences in Bilingual Corpora. In Proceedings of ACL-1991. our card
  11. T. Gollins and M. Sanderson. 2001. Improving cross language retrieval with triangulated translation. In SIGIR. find
  12. Raymond G. Gordon, Jr., editor. 2005. Ethnologue: Languages of the World (Fifteenth Edition). SIL International. find
  13. A. Haghighi, P. Liang, T. Berg-Kirkpatrick, and D. Klein. 2008. Learning bilingual lexicons from monolingual corpora. In ACL. our card
  14. S. Helmreich, L. Guthrie, and Y. Wilks. 1993. The use of machine readable dictionaries in the Pangloss project. In AAAI Spring Symposium on Building Lexicons for Machine Translation. find
  15. Monika R. Henzinger, Allan Heydon, Michael Mitzenmacher, and Marc Najork. 1999. Measuring index quality using random walks on the web. In WWW. find
  16. R. Hoffmann, S. Amershi, K. Patel, F. Wu, J. Fogarty, and D. S. Weld. 2009. Amplifying community content creation with mixed-initiative information extraction. In ACM SIGCHI (CHI2009). find
  17. D. R. Karger. 1999. A randomized fully polynomial approximation scheme for the all-terminal network reliability problem. SIAM Journal of Computation, 29(2):492–514. find
  18. K. Knight and S. Luk. 1994. Building a large-scale knowledge base for machine translation. In AAAI. find
  19. I.D. Melamed. 1997. A Word-to-Word Model of Translational Equivalence. In Proceedings of ACL-1997 and EACL-1997, pages 490–497. find
  20. M. Neff and M. McCord. 1990. Acquiring lexical data from machine-readable dictionary resources for machine translation. In 3rd Intl Conference on Theoretical and Methodological Issues in Machine Translation of Natural Language. find
  21. P. Vossen, editor. 1998. EuroWordNet: A multilingual database with lexical semantic networds. Kluwer Academic Publishers. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mausam-2009-panlingual-dictionary/