Elamur Research⌕
☰
home / papers / 2014 / Pivot-based multilingual dictionary building using Wiktionary

Pivot-based multilingual dictionary building using Wiktionary

Summary

The paper addresses the shortage of machine-readable bilingual dictionaries, which barely exist for most language pairs. The author extends them with triangulation: if two words from different languages translate the same word of a third language (the pivot), they are probably translations of each other. The data come from the translation tables of 43 Wiktionary editions (February 2014 dumps), parsed with wikt2dict, the author's own open-source tool. In total, 53 languages are covered, including Ukrainian. The main obstacle is noise from polysemous pivot words, editor errors and parsing failures. The Inverse Consultation filter accepts only pairs found through at least two pivots, so it discards too much. The author instead describes each candidate with "edge diversity" features in the translation graph and trains a maximum entropy classifier. The training data were collected automatically, without manual annotation. The classifier reaches an F1 of 0.9347, and coverage of the basic vocabulary rises from about 67% to over 80%. The resulting dictionaries are freely available, which is especially valuable for low-resource languages.

Key points

  • 4.09 million pairs were extracted directly from Wiktionary. Triangulation yielded 32.55 million candidates, 29.6 million of them new. Only 7.63 million were found through two or more pivots. The classifier accepted 19.39 million candidates as correct.

  • The features take into account:

    • the number of pivot languages and pivot words;
    • the number of languages and edges on the left and right sides of the triangle;
    • the source Wiktionary editions the edges were taken from.

    They were also computed separately for each language, giving over 2000 features. Reducing them to 200 did not lower quality: F1 0.9347 versus 0.9345 with the full set.

  • Training data: 1.77 million positive examples. Negative examples came from two anomaly filters: a punctuation filter (320 thousand) and a character unigram filter (70 thousand).

  • Coverage (recall) was measured on the 4lang concept lexicon of about 3,500 basic words. Wiktionary alone covers 67.4%, together with triangles 82.4%, and after the classifier 80.2%. For the 10 best-covered languages, it reaches about 95%.

  • Manual evaluation was carried out for 10 language pairs, with 100 candidates each. Wiktionary pairs are correct 88.4% of the time on average. New triangulation pairs are correct only 47.9% of the time, and 34.7% of them are wrong.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4twce9t7r8ga

@inproceedings{acs2014pivot,
  title = {{Pivot-based multilingual dictionary building using Wiktionary}},
  author = {Acs, Judit},
  booktitle = {LREC 2014 (Reykjavik)},
  year = {2014},
  doi = {10.63317/4twce9t7r8ga},
  url = {https://doi.org/10.63317/4twce9t7r8ga}
}

References

9 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ács, J., Pajkossy, K., and Kornai, A. (2013). Building basic vocabulary across 40 languages. In Proceedings of the Sixth Workshop on Building and Using Comparable Corpora, pages 52–58, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  2. Berners-Lee, T. (2009). Linked data. W3C design document http://www.w3.org/DesignIssues/LinkedData.html. ↗
  3. Kaji, H., Tamamura, S., and Erdenebat, D. (2008). Automatic construction of a Japanese-Chinese dictionary via English. In LREC, volume 2008, pages 699–706. find
  4. Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P. N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., et al. (2013). DBpedia - a large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web Journal (under review, 2013). find
  5. Saralegi, X., Manterola, I., and Vicente, I. S. (2011). Analyzing methods for improving precision of pivot based bilingual dictionaries. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 846–856. Association for Computational Linguistics. find
  6. Saralegi, X., Manterola, I., and Vicente, I. S. (2012). Building a Basque-Chinese dictionary by using English as pivot. In Chair), N. C. C., Choukri, K., Declerck, T., Doğan, M. U., Maegaard, B., Mariani, J., Odijk, J., and Piperidis, S., editors, Proceedings of the Eight International Conference o… find
  7. Soderland, S., Etzioni, O., Weld, D. S., Skinner, M., Bilmes, J., et al. (2009). Compiling a massive, multilingual dictionary via probabilistic inference. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Proce… find
  8. Tanaka, K. and Umemura, K. (1994). Construction of a bilingual dictionary intermediated by a third language. In Proceedings of the 15th conference on Computational linguistics-Volume 1, pages 297–303. Association for Computational Linguistics. find
  9. Zesch, T., Müller, C., and Gurevych, I. (2008). Extracting lexical semantic knowledge from wikipedia and wiktionary. In LREC, volume 8, pages 1646–1652. find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/acs-2014-wikt2dict-pivot/