📄 Language Models are Few-shot Multilingual Learners

Summary

The paper asks whether large language models trained mostly on English can perform multi-class intent classification in other languages through in-context learning alone, with no parameter updates. The authors test GPT-2, GPT NEO, GPT NEO-J and T5 on SNIPS, MTOP (English, French, German, Spanish) and MultiNLU (English, Spanish). They examine both monolingual prompts and a cross-lingual setup, where the English examples in the context differ from the language of the query. Their method turns each intent class into a separate binary question. The prompt holds positive and negative examples for that class, and the model's normalized probability of answering "true" serves as a confidence score. The class with the highest score wins. Accuracy rises well above chance as more shots are added and as model size grows. GPT models do better than T5. In the cross-lingual MultiNLU setting, GPT NEO-J nearly matches fine-tuned and translation-based systems. This suggests that gradient-free prompting is a cheap option for cross-lingual NLU when labeled data or compute is scarce.

Key points

• Data: SNIPS (English), MTOP (en/fr/de/es) and MultiNLU (en/es). Results are reported as mean accuracy with standard deviation over three runs.
• Models and limits: GPT-2, GPT NEO, GPT NEO-J and T5 were evaluated. Context limits shaped the prompt design: 1,024 tokens for GPT-2 XL and 2,048 for GPT NEO.
• Cross-lingual result: On MultiNLU (en→es), GPT NEO-J trails the fine-tuned state-of-the-art baseline of Liu et al. (2020b) by about 4.2%. It trails the Translate-Train model by under 3%.
• Language pairs: On MTOP, en→es transfer generally works better than en→de or en→fr. T5 lags behind GPT models, possibly because of how it was pre-trained.
• Baselines and code: The baselines were zero-shot, zero-shot cross-task (entailment-style scoring) and fine-tuning. Code is available at github.com/gentaiscool/few-shot-lm.

Interesting ideas

• I1. Per-class confidence scoring instead of free generation: Each class gets its own binary true/false query, and the normalized probability of "true" is compared across classes. This avoids the model producing unexpected words. It could fit any task with a closed label set.
• I2. One prompt per class: Building a separate short prompt for each label avoids packing every class's examples into one context window. The prompts can also be run in parallel. This is helpful when there are many labels or the context length is limited.
• I3. Prompt in one language, query in another: Using an English context with a target-language query is a cheap probe of cross-lingual ability. It could be used to check how well a mostly-English model handles another language before investing in annotation.
• I4. Order the examples within the prompt: Placing positive examples before negative ones gave better results than random shuffling, especially for T5. Measuring variance across shuffled orders is a simple robustness check for any prompting setup.
• I5. Study how token overlap affects transfer: The authors suggest correlating transfer quality with token and named-entity overlap between source and target languages. They note that English tokenizers handle non-Latin scripts poorly. This analysis is directly relevant when judging whether such models would work for languages written in Cyrillic or other scripts.