📄 A Corpus for Multilingual Document Classification in Eight Languages

Summary

Cross-lingual document classification trains a classifier in one language and applies it to another without labelled data in the target language. Until this paper, the standard evaluation used a portion of RCV2 (Reuters Corpus Volume 2) limited to English, German, French and Spanish, and most work studied only English↔German. The authors also found that class frequencies in these test sets differed widely between languages. Because of this, scores partly reflected how well a system handled class imbalance rather than how well it transferred across languages.

To fix this, they built MLDoc from RCV2. It covers eight languages, with the same number of documents per class and language, and fixed train, development and test splits. They report baselines for every transfer direction using two methods: MultiCCA word embeddings combined by a convolutional network, and joint multilingual sentence embeddings learned with machine-translation-style encoders. They evaluate zero-shot, targeted and joint multilingual settings. The sentence embeddings generally transfer more robustly, while the word-embedding system works best mainly when trained on English. MLDoc gives the field a shared, balanced benchmark for many language pairs.

Key points

• Source data. RCV2 has about 487,000 non-parallel news stories in 13 languages. Each story is labelled with one of four top categories: CCAT, ECAT, GCAT and MCAT.
• Problem with the old subset. In the earlier English/German test sets, over 80% of documents were GCAT or MCAT and at most 2% were CCAT. In the French and Spanish test sets, CCAT made up about 21% and 15%.
• MLDoc contents.
  • Languages: DE, EN, ES, FR, IT, RU, ZH and JA.
  • Training sets of 1k, 2k, 5k and 10k stories, a 1k development set and a 4k test set, all class-balanced.
  • Spanish and Russian cannot reach the full 10k (9,458 and 5,216 documents available).
  • The corpus itself cannot be redistributed, so extraction scripts are published at github.com/facebookresearch/MLDoc.
• Baselines.
  • Word-level: MultiCCA embeddings aggregated by a CNN.
  • Sentence-level: BiLSTM + max-pooling joint sentence embeddings, trained on Europarl (EN, DE, FR, ES, IT) or on the United Nations corpus (EN, FR, ES, RU, ZH), with a one-hidden-layer MLP as the classifier.
• Results.
  • In zero-shot transfer, the English-trained MultiCCA model scores best on DE, IT and ZH (e.g. 81.2 on German). The sentence embeddings score best on EN, ES, FR and RU.
  • Joint training on 200 examples from each of five languages gives the highest averages: 83.31 (MultiCCA) and 82.04 (sentence embeddings).

Interesting ideas

• I1. Balance the classes in test sets. Sampling equal numbers per class in every language keeps label-distribution shifts from hiding genuine transfer quality. This applies to any multilingual benchmark, including one built for Ukrainian.
• I2. Name the transfer settings explicitly. The paper separates zero-shot (source-language dev set only), targeted (target-language dev set used for model selection) and joint (a small mixed-language training set). Stating which setting was used makes results comparable and shows how much a little target-language data helps.
• I3. Test all language pairs, not just pairs with English. Training on each language and testing on all the others revealed that some systems depend heavily on English as the source. In this study, German or French sometimes turned out to be better sources.
• I4. Ship extraction tools instead of data. When a license blocks redistribution, publishing deterministic scripts that rebuild the exact subsets still gives everyone a reproducible benchmark.
• I5. Spread a fixed labelling budget across languages. At the same total size, a training set mixing several languages beat single-language training for transfer. This is worth considering when deciding how to spend annotation effort.