Elamur Research⌕
☰
home / papers / 2021 / Cross-lingual Contextualized Topic Models with Zero-shot Learning

Cross-lingual Contextualized Topic Models with Zero-shot Learning

Paper · 2021 · Federico Bianchi, Silvia Terragni, Dirk Hovy et al.

Summary

Classic bag-of-words topic models are bound to the vocabulary of their training language. They cannot process unseen words, and they cannot be reused for texts in other languages. Bianchi et al. propose ZeroShotTM, a neural topic model built on Neural-ProdLDA (a VAE). It takes multilingual SBERT document embeddings as input in place of the BoW. The BoW is still reconstructed by the decoder during training, which supplies the topic words, but it is not needed at inference. As a result, a model trained only on English can assign topics to documents in languages it never saw, with topic descriptors in English. The authors use DBpedia abstracts from English Wikipedia and test on 300 abstracts with counterparts in Italian, French, Portuguese and German. On English data, the contextual input keeps coherence on par with BoW-based models. Cross-lingually, predicted topics usually match those of the English version, roughly as well as a machine-translation pipeline. This makes topic modeling possible for languages with little data and requires no alignment resources.

Key points

  • Data: W1 contains 20,000 English abstracts. W2 contains 100,000 abstracts, split into 99,700 for training and 300 for testing, with comparable test documents in IT, FR, PT and DE. The BoW vocabulary uses the 2,000 most frequent non-stopwords.
  • Coherence on W1 (NPMI, 50/100 topics): ZeroShotTM scores 0.1632/0.1381, comparable to Combined TM (0.1644/0.1409) and Neural-ProdLDA (0.1658/0.1285). LDA scores negative.
  • Zero-shot topic matches with English: ZeroShotTM averages 78.00% with 25 topics and 64.41% with 50. The DeepL translation baseline (Ori) reaches 76.00% and 69.00%, while a uniform baseline reaches 4% and 2%. Centroid similarity is 0.85 and 0.77, and KL divergence is 0.15 and 0.20.
  • Manual evaluation: 1,500 documents were rated, with an average topic quality of 2.26 out of 3 and inter-rater agreement of Gwet AC1 = 0.88.
  • Code release: the model is available as the Python package contextualized-topic-models (MilaNLProc on GitHub). It uses the SBERT model distiluse-base-multilingual-cased.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Bocconi UniversityMilan · Italy16 works · 2,850 citations🎓University of Milano-BicoccaMilan · Italy10 works · 4,918 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza and Elisabetta Fersini. 2021. Cross-lingual Contextualized Topic Models with Zero-shot Learning. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.eacl-main.143

@inproceedings{bianchi2021cross,
  title = {{Cross-lingual Contextualized Topic Models with Zero-shot Learning}},
  author = {Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza and Elisabetta Fersini},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.143},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.143}
}

References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2020. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. arXiv preprint arXiv:2004.03974.
  2. David M Blei. 2012. Probabilistic topic models. Communications of the ACM, 55(4):77–84. find
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022.
  4. Jordan L. Boyd-Graber and David M. Blei. 2009. Multilingual topic models for unaligned text. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, UAI 2009, pages 75–82. AUAI Press. find
  5. Jordan L. Boyd-Graber, David Mimno, and David Newman. 2014. Care and Feeding of Topic Models: Problems, Diagnostics, and Improvements, CRC Handbooks of Modern Statistical Methods. CRC Press. find
  6. Stephen Carrow. 2018. Pytorchavitm: Open source avitm implementation in pytorch. Github. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  8. Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439–453.
  9. E.D. Gutiérrez, Ekaterina Shutova, Patricia Lichtenstein, Gerard de Melo, and Luca Gilardi. 2016. Detecting cross-cultural differences using a multilingual topic model. Transactions of the Association for Computational Linguistics, 4:47–60. find
  10. Kilem L Gwet. 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC. find
  11. Shudong Hao and Michael J. Paul. 2018. Learning multilingual topics from incomparable corpora. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, pages 2595–2609. Association for Computational Linguistics. find
  12. Geert Heyman, Ivan Vulic, and Marie-Francine Moens. 2016. C-BiLDA extracting cross-lingual topics from non-parallel texts by distinguishing shared from unshared content. Data Mining and Knowledge Discovery, 30(5):1299–1323. find
  13. Dirk Hovy, Federico Bianchi, and Tommaso Fornaciari. 2020. “You sound just like your father” Commercial machine translation systems include stylistic biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1686–1690, Online. Association for Computati… find
  14. Dirk Hovy and Christoph Purschke. 2018. Capturing regional variation with distributed place representations and geographic retrofitting. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4383–4394, Brussels, Belgium. Association for Computational Lingui… find
  15. Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg…
  16. Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, and Zongyang Ma. 2016. Topic modeling for short texts with auxiliary word embeddings. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, July 17-21, 2016, pa… find
  17. Xiaodong Liu, Kevin Duh, and Yuji Matsumoto. 2015. Multilingual topic models for bilingual dictionary extraction. ACM Transactions on Asian Language Information Processing, 14(3):11:1–11:22. find
  18. Tengfei Ma and Tetsuya Nasukawa. 2017. Inverted bilingual topic models for lexicon extraction from non-parallel data. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI 2017, pages 4075–4081. IJCAI.org. find
  19. David M. Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual topic models. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, pages 880– 889. Association for Computational Linguistics. find
  20. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making Sense of Language-Specific BERT Models. arXiv preprint arXiv:2003.02912. — [arXiv:2003.02912](https://arxiv.org/abs/2003.02912) find
  21. Debora Nozza, Elisabetta Fersini, and Enza Messina. 2016. Unsupervised irony detection: a probabilistic model with word embeddings. In International Conference on Knowledge Discovery and Information Retrieval, volume 2, pages 68–76. SCITEPRESS. find
  22. Jagadeesh Jagarlamudi and Hal Daumé. 2010. Extracting multilingual topics from unaligned comparable corpora. In Advances in Information Retrieval, pages 444–456, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  23. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3973–3983.…
  24. Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014. find
  25. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. arXiv preprint arXiv:2002.12327.
  26. Kriste Krstovski, David A. Smith, and Michael J. Kurtz. 2016. Online multilingual topic models with multi-level hyperpriors. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan-Akash Srivastava and Charles A. Sutton. 2017. A… find
  27. Silvia Terragni, Elisabetta Fersini, and Enza Messina. 2020a. Constrained relational topic models. Information Sciences, 512:581–594. find
  28. Silvia Terragni, Debora Nozza, Elisabetta Fersini, and Messina Enza. 2020b. Which matters most? comparing the impact of concept and document relationships in topic models. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 32–40, Online. Association for Computationa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bianchi-2021-cross-lingual-contextualized-topic-models/