📄 Polylingual Topic Models

Summary

Topic models had so far been used mostly on single-language collections, or at most on language pairs. Existing bilingual variants were also built for machine translation and depended on sentence-aligned parallel text. Mimno et al. propose the polylingual topic model (PLTM), an extension of LDA. In PLTM, a "tuple" of loosely equivalent documents in different languages shares one topic distribution, while each topic has a separate word distribution for every language. Inference uses Gibbs sampling, and the hyperparameters are re-estimated during training.

The model is tested on two corpora:

• EuroParl, with 11 languages aligned at the speech level, which checks behaviour on true translations.
• Wikipedia, with 12 languages, where linked articles are comparable but not translations.

PLTM recovers clearly aligned topics in every language, covering both technical acronyms and everyday vocabulary. Linked Wikipedia articles turn out to be topically consistent on average across languages, yet each language emphasises topics differently overall. This matters because comparable multilingual data such as Wikipedia is far more plentiful than parallel corpora. The model also helps with lexicon induction for low-resource pairs and with cross-lingual trend analysis.

Key points

• EuroParl setup: 11 Western European languages, roughly a decade of parliamentary proceedings, and more than 121 million words. The 25 most frequent word types were removed. Lexical variety per 10,000 tokens differs widely, for example Finnish at 336.2 versus English at 43.1.
• Wikipedia setup: 12 languages, namely English (EN), German (DE), French (FR), Italian (IT), Polish (PL), Russian (RU), Turkish (TR), Greek (EL), Finnish (FI), Hebrew (HE), Persian (FA) and Welsh (CY).
• Training settings: 1000 Gibbs iterations, β set to 0.01 for each language, and α initialised to 0.01·T with a uniform base measure. αm is re-optimised every 10 iterations, and the example topics come from T=400.
• Cross-language divergence: average Jensen-Shannon divergence between language pairs on Wikipedia falls mostly in the 0.08–0.12 range at T=400. This matches EuroParl, even for language pairs with a history of conflict.
• Uneven emphasis: topic emphasis differs by language. Cross-country skiing is prominent in Finnish but almost absent in Welsh, while topics about actors are uniform across languages.

Interesting ideas

• I1. Share the mixture, separate the vocabularies: a single topic mixture per tuple combined with per-language word distributions avoids explicit word alignment. The design scales easily past two languages and could suit Ukrainian–Russian–English comparable corpora or multilingual trade documents.
• I2. Translations as a controlled testbed: the authors first run the model on genuine translations, where the shared-topic assumption is known to hold, and only then move to noisier comparable data. This is a sensible validation step for any cross-lingual method.
• I3. Per-language divergence inside a tuple: splitting each tuple's topic assignments by language and computing the JS divergence between the parts measures how much versions of the "same" document disagree. It could flag Wikipedia articles whose coverage differs across languages.
• I4. Language-level topic shares: counting the proportion of tokens per topic in each language reveals cultural or regional focus, which is useful for geography-oriented or media comparison studies.
• I5. Topics as a lexicon seed: aligned topics provide candidate translation pairs without parallel sentences. These candidates can feed heavier alignment methods for low-resource language pairs.