🌍 Multilingual & cross-lingual NLP

A survey · about 20 minutes to read · every paper named here has its own page in this library

Multilingual and cross-lingual natural language processing studies how to build language technology that works for many languages at once, and how to carry what a system has learned in one language over to another. Roughly seven thousand languages are spoken in the world, yet annotated data, large corpora and evaluation sets exist for only a small fraction of them, and for most of the history of the field the best-resourced language by far has been English. Multilingual NLP asks whether one model, one annotation scheme or one benchmark can serve dozens or hundreds of languages; cross-lingual NLP asks how labels, lexicons, parsers or classifiers built for a resource-rich language can be reused where such resources are missing.

The topic sits at a crossroads. It borrows from machine translation the habit of thinking in terms of parallel text and alignment, from linguistic typology the awareness that languages differ systematically in word order, morphology and script, and from representation learning the idea that words and sentences of different languages can live in one shared vector space. Over three decades the dominant tool changed several times: projection of annotations through word-aligned bitexts, cross-lingual word embeddings, encoders pretrained jointly on many languages (multilingual BERT, later XLM-R), massively multilingual translation systems, and finally large generative language models that are prompted in many languages.

The papers here follow that whole arc, from the parsing shared tasks of the 2000s to recent studies of large language models outside English.

What it is

Multilingual versus cross-lingual

The two adjectives are often used together but mean different things. A multilingual system or resource covers several languages, either with one shared model or with a common annotation framework applied to each language separately. A cross-lingual method moves knowledge across a language boundary: a classifier trained on English reviews is applied to Chinese ones, a parser trained on Czech is used for Slovak, or a query in one language retrieves documents written in another. Much modern work is both at once: a single multilingual encoder is fine-tuned on labelled data in one language and then evaluated, without further training, on others.

Core notions

• Zero-shot cross-lingual transfer. A model is trained on a task using labels from a source language only (usually English) and tested directly on target languages. This setting became the standard evaluation for multilingual encoders.
• Few-shot transfer and target adaptation. A small number of labelled target-language examples, or continued pre-training on unlabelled target text, is added on top of zero-shot transfer.
• Translate-train and translate-test. Instead of relying on shared representations, machine translation is used either to translate the training data into the target language or to translate test inputs into the source language. These remain strong baselines.
• Annotation projection. Labels on one side of a parallel corpus are carried over to the other side through word alignments, producing noisy training data for the second language.
• Shared representation spaces. Word or sentence vectors of different languages are placed in a common space, so that translations end up near each other and a model working on vectors can ignore which language they came from.
• The curse of multilinguality. With fixed capacity, adding more languages to one model first helps low-resource languages through transfer but eventually degrades performance on all of them; balancing transfer against interference is a recurring design problem.

How it differs from its neighbours

Machine translation is the closest neighbour, and many multilingual techniques (shared encoders, language tokens, back-translation) were invented for translation. The difference is in the goal: translation produces text in another language, while multilingual NLP mostly aims at analysis or understanding tasks — tagging, parsing, classification, question answering, retrieval — in many languages. Low-resource NLP overlaps heavily, since cross-lingual transfer is a main route to tools for under-served languages, but low-resource work also includes data collection and documentation for single languages.

History and milestones

Before the neural era: parallel text and harmonised resources

Large parallel corpora built from institutional translations — parliamentary proceedings, United Nations documents and the body of European Union law — supplied the bitext from which both translation systems and cross-lingual methods learned. The JRC-Acquis collection, compiled from EU legislation in over twenty languages, and MultiUN are representative. Projects such as SIMPLE tried to define semantic lexicons on a common model for many European languages, and the MATE meta-scheme did the same for coreference annotation in dialogue.

A conceptual turning point came with Yarowsky, Ngai and Wicentowski (2001), who showed that English tools could be "projected" through word-aligned bitext to induce taggers, chunkers and lemmatisers for French, Chinese, Czech and Spanish. Projection, together with robust training on noisy labels, became the template for a decade of cross-lingual work in syntax, semantics and sentiment.

Shared tasks across languages (2006–2012)

The CoNLL shared tasks pushed parsing beyond English. The CoNLL-X task of 2006 converted thirteen treebanks into one column format and scored every system with one metric, and CoNLL 2007 extended the comparison to ten typologically varied languages plus domain adaptation. The CoNLL-2009 task joined syntactic dependencies with semantic roles in seven languages, and CoNLL-2012 applied the same idea to coreference, using OntoNotes data for Arabic, Chinese and English. They revealed that annotation conventions, not only the languages, explained much of the variation in scores — a lesson that led to harmonised annotation.

In the same period, delexicalised and cluster-based transfer and selective parameter sharing showed that parsers could be moved between languages without target treebanks, and related-language parser adaptation exploited closeness between Danish and Swedish. Cross-lingual sentiment analysis grew out of the same idea: Mihalcea, Banea and Wiebe built Romanian subjectivity tools from English ones, and Wan's co-training method used machine translation to treat English and Chinese versions of a review as two views of the same example.

Cross-lingual embeddings (2012–2018)

Distributed word representations made it possible to share meaning across languages in a continuous space. Klementiev, Titov and Bhattarai (2012) trained coupled neural language models so that likely translations got similar vectors; Faruqui and Dyer (2014) used canonical correlation analysis on translation pairs; Hermann and Blunsom learned compositional sentence representations from parallel data; and Ammar and colleagues placed words from dozens of languages in a single space using only dictionaries and monolingual text. Mapping-based methods then removed the need for any bilingual signal: adversarial training and the self-learning VecMap method of Artetxe, Labaka and Agirre learned a mapping between two embedding spaces from monolingual data alone.

Universal Dependencies arrived in the same years. The first UD release paper fixed a single set of part-of-speech tags, morphological features and dependency relations for all languages, and the CoNLL 2017 and CoNLL 2018 shared tasks asked systems to parse raw text into UD trees for dozens of languages. A single parser for many languages, as in Many Languages, One Parser, became realistic.

Multilingual neural translation and sentence encoders (2016–2019)

Neural translation showed that one network could serve many language pairs. Firat, Cho and Bengio (2016) shared one attention module among many encoders and decoders, and Ha, Niehues and Waibel went further with a single universal encoder and decoder. Google's work on massively multilingual translation in the wild and Aharoni, Johnson and Firat scaled this to around a hundred languages and made the tension between positive transfer and interference explicit. Translation models also turned out to be good sentence encoders: Schwenk and Douze and then LASER used the encoder of a many-language translation system as a language-agnostic sentence embedder.

Multilingual pretrained encoders (2019–2021)

Multilingual BERT, released in late 2018 and trained on Wikipedia in about a hundred languages with no parallel data, changed the field almost overnight. Two papers from 2019 — How Multilingual is Multilingual BERT? and Beto, Bentz, Becas — showed that it transferred surprisingly well across languages, even across scripts. A wave of analysis followed, asking why: Artetxe, Ruder and Yogatama showed that shared vocabulary and joint training were not strictly necessary, and Emerging Cross-lingual Structure in Pretrained Language Models found that parameter sharing mattered more than shared anchors. XLM and then XLM-R, trained on far larger web corpora, became the strongest encoders of the period. Benchmarks such as XTREME, XGLUE, TyDi QA, MLQA and XCOPA gave the community shared yardsticks.

Large language models (2021 onwards)

Generative models trained mainly on English proved able to work in other languages, but unevenly. XGLM and mGPT trained decoder models with deliberately balanced language mixes; BLOOMZ and mT0 showed that multitask instruction tuning generalises across languages; the Aya model and Aya dataset built instruction data in over a hundred languages through a participatory effort. Evaluations of ChatGPT — Bang and colleagues, Lai and colleagues and MEGA — documented sharp drops for low-resource and non-Latin-script languages. Most papers on this topic in the library date from 2018 or later, and the most recent years are the busiest.

Main problems and tasks

Transfer for structured prediction

Part-of-speech tagging, named entity recognition, dependency parsing and semantic role labelling were the first tasks for which cross-lingual transfer was studied systematically, because harmonised datasets existed. Typical work here ranges from bidirectional LSTM taggers across dozens of languages and multilingual constituency parsing to NER transfer from many source languages and cross-lingual NER with minimal resources. Name tagging and linking for all Wikipedia languages produced the WikiANN data that later benchmarks reused.

Understanding, reasoning and question answering

Inference (XNLI), paraphrase (PAWS-X), causal commonsense (XCOPA), extractive QA (MLQA, XQuAD, TyDi QA), open-retrieval QA (XOR QA) and grade-school arithmetic in MGSM test whether models understand rather than merely tag text in many languages.

Sentence similarity, retrieval and bitext mining

Semantic textual similarity moved from English into Spanish and cross-lingual pairs in the SemEval-2014 and SemEval-2016 tasks. Multilingual sentence encoders made cross-lingual retrieval practical and turned into tools for mining parallel sentences from the web (margin-based mining, corpus filtering). Multilingual retrieval benchmarks such as MIRACL and embedding families such as multilingual E5 and mGTE extend this line.

Social media, sentiment and harmful content

Many applied datasets are multilingual by design: hate speech in English and Spanish (HatEval) and in English, French and Arabic (Ousidhoum and colleagues), the Hurtlex lexicon of offensive words, Twitter models such as XLM-T, and sentiment corpora for Nigerian languages (NaijaSenti).

Code-switching

Multilingual speakers often mix languages within one utterance, and this is especially common on social media in South Asia. Datasets for English–Hindi POS tagging, Tamil–English and Malayalam–English sentiment, and the benchmarks GLUECoS and LinCE treat mixed-language text as a task in its own right.

Speech

Speech technology faces the same coverage problem. Corpora such as Common Voice, Multilingual LibriSpeech, VoxPopuli and MuST-C and models such as XLS-R and SeamlessM4T bring multilingual pretraining to audio.

Discourse, coreference and generation

Higher-level tasks lag behind. Multilingual coreference has been the subject of the SemEval-2010 task, CoNLL-2012 and, more recently, the CRAC shared tasks built on harmonised corpora (second edition findings); zero anaphora is a recurring cross-lingual problem. Discourse parsing (CoNLL 2016), summarisation (MEAD, MLSUM, WikiLingua) and surface realisation (SR'18) round out the picture.

Methods and schools

Projection and resource transfer

The oldest family of methods moves annotations rather than models. Labels produced by an English tool are carried through word alignments onto a parallel text and used to train a new tool, as in the 2001 projection work, the proposition banks for semantic roles generated by filtered projection, and multi-source projection for parsing truly low-resource languages. Lexicons can be translated instead, as was done for subjectivity clues. Its weakness is noise from misalignment and from real divergence between languages.

Direct model transfer

A model trained on the source language is applied to the target language unchanged, using features that do not depend on the word forms of either: universal POS tags, cross-lingual word clusters, distributed representations or aligned contextual embeddings. Choosing the right source language — by typological closeness, data size or lexical overlap — matters a great deal.

Shared embedding spaces

Cross-lingual word embeddings can be learned jointly from parallel data, or learned separately for each language and then mapped together with a linear, often orthogonal, transformation. The mapping view, refined by unsupervised initialisation and self-learning, dominated around 2017–2018. Adversarial domain adaptation offered a complementary route: ADAN trains features that predict sentiment while hiding the input language from a discriminator.

Joint multilingual pretraining

The dominant paradigm since 2019 trains one transformer on concatenated monolingual corpora of many languages with a shared subword vocabulary, optionally adding parallel data through translation language modelling (Unicoder, InfoXLM). Debates in this school centre on why transfer works at all, how to sample languages, how much capacity each language needs (larger-scale multilingual MLM), and how to handle negative interference. Tokenisation is part of the story: Rust and colleagues showed that much of the gap between multilingual and monolingual models comes from vocabularies that split words of some languages into too many pieces, and Chung and colleagues reconsidered how input and output embeddings are tied.

Modular and adaptive approaches

Because a frozen multilingual model spreads its capacity thinly, modular methods add small language-specific components. MAD-X stacks language and task adapters; UNKs Everywhere adapts models to unseen scripts; modular pre-training builds language-specific capacity in from the start. Continued pre-training on a target language, as in multilingual adaptive fine-tuning for African languages or continual pre-training for Japanese, is a simpler alternative.

Monolingual versus multilingual models

A persistent debate asks whether a language is better served by its own model. Dedicated encoders such as BERTje for Dutch, FinBERT for Finnish and AraBERT for Arabic often beat multilingual BERT on their language, while regional models such as IndicBERT and MuRIL share capacity among related languages. Small Data? No Problem! showed that a compact model trained only on African languages can compete with much larger ones.

Translation as a component

Machine translation remains a strong baseline in every task, either by translating training data into the target language or test data into the source. It also leaves traces: Translation Artifacts in Cross-lingual Transfer Learning showed that benchmarks built by translation reward systems in subtle ways, which motivated datasets written natively in each language.

Prompting large language models

With generative models the questions shift to prompting: in which language to write instructions and demonstrations, whether to let a model reason in English, and how to elicit cross-lingual thought or cross-lingual chain-of-thought. Instruction tuning in many languages (Okapi, Aya 23, Aya Expanse) and regional models such as EuroLLM aim to reduce the dominance of English.

Datasets, resources and benchmarks

A lesson repeated across these resources is that size is not quality. The audit of web-crawled multilingual datasets found that many low-resource portions of popular corpora contained little or no text in the labelled language, and benchmark designers increasingly prefer data written natively rather than translated from English.

Landmark papers in this library

How Multilingual is Multilingual BERT? (Pires, Schlinger and Garrette, 2019). The most cited work on this topic here. By fine-tuning multilingual BERT on NER or POS tagging in one language and testing in another, it showed that transfer succeeds even with almost no shared word pieces and across different scripts, while it suffers when word order differs and when text is transliterated.

Universal Dependencies v1 (Nivre et al., 2016). The reference description of the first UD release: lexicalist dependency annotation with content words as heads, one inventory of universal tags, features and relations, the CoNLL-U format and validation tools. UD became the default resource for multilingual parsing and morphology, and its version 2 description documents its growth.

CoNLL-X Shared Task on Multilingual Dependency Parsing (Buchholz and Marsi, 2006). It established the common column format, a single attachment metric and equal-sized test sets for thirteen languages, so that parsers could finally be compared across languages on identical terms.

Language-agnostic BERT Sentence Embedding (Feng et al., 2020). LaBSE combined masked-language pre-training with a dual-encoder translation ranking objective and produced sentence vectors for over a hundred languages that markedly improved bitext retrieval over LASER.

Multilingual sentence embeddings by knowledge distillation (Reimers and Gurevych, 2020). An English sentence-embedding model acts as a teacher, and a multilingual student learns to map both a sentence and its translation to the teacher's vector. The approach separates the cost of building a good English model from the cost of adding languages and is a standard recipe in practice.

On the Cross-lingual Transferability of Monolingual Representations (Artetxe, Ruder and Yogatama, 2020). By teaching an English-only BERT new lexical embeddings for a second language while keeping its body frozen, the authors matched jointly trained multilingual models on several benchmarks. The result challenged the view that shared subwords are what make transfer work, and the paper introduced the XQuAD dataset.

LASER: massively multilingual sentence embeddings (Artetxe and Schwenk, 2018). LASER trained a BiLSTM encoder as part of a translation system covering over ninety languages and reused it as a universal sentence embedder; it also introduced the Tatoeba similarity-search benchmark. It became a workhorse for cross-lingual classification and bitext mining.

VecMap: robust unsupervised mapping of word embeddings (Artetxe, Labaka and Agirre, 2018). The VecMap method builds a weak seed dictionary from the shape of similarity distributions and then refines it iteratively, succeeding on distant pairs such as English–Finnish where adversarial methods had failed.

MAD-X (Pfeiffer, Vulić, Gurevych and Ruder, 2020). Language adapters and task adapters, plus an invertible adapter on the embeddings, let a frozen XLM-R be specialised cheaply for languages absent from pre-training. It made modular adaptation a mainstream idea.

Multi-way multilingual NMT with shared attention (Firat, Cho and Bengio, 2016). The paper demonstrated that attention could be shared across language pairs, keeping model growth linear in the number of languages and helping data-poor pairs most.

Inducing text analysis tools by robust projection (Yarowsky, Ngai and Wicentowski, 2001). The founding paper of annotation projection, with noise-tolerant training that turned very imperfect projected labels into usable taggers, bracketers and lemmatisers without any hand-annotated target data.

XTREME (Hu et al., 2020). Nine tasks in forty languages, evaluated in the zero-shot setting, gave multilingual encoders a GLUE-style leaderboard. Its analysis showed weak transfer for syntax and retrieval and for languages far from English, and it was later revised as XTREME-R.

TyDi QA (Clark et al., 2020). Native speakers wrote questions they genuinely wanted answered, in eleven typologically diverse languages, without translation. The design avoided the artefacts of translated benchmarks and became a model for later dataset construction.

XLS-R (Babu et al., 2021). Scaling self-supervised wav2vec 2.0 pre-training to over a hundred languages of public speech data, the authors showed large gains in speech translation, recognition and language identification, and that a big enough cross-lingual model can rival English-only pre-training.

Crosslingual Generalization through Multitask Finetuning (Muennighoff et al., 2022). The BLOOMZ and mT0 models showed that instruction tuning on English data already improves zero-shot behaviour in other languages, and that the xP3 multilingual mixture helps further.

Language Models are Multilingual Chain-of-Thought Reasoners (Shi et al., 2022). The MGSM benchmark, built by professional translation of arithmetic word problems, showed that multi-step reasoning in many languages appears only in very large models, and that reasoning in English is often as good as reasoning in the question's language.

People

• Sebastian Ruder is the most prolific author on this topic in the library. His papers here span the analysis of transfer (monolingual representations), benchmarks (XTREME, XTREME-R), adapters (MAD-X) and multilingual reasoning; his affiliations include the University of Edinburgh, MIT and the University of Washington.
• Daniel Zeman of Charles University led the CoNLL 2017 and 2018 shared tasks on parsing raw text into Universal Dependencies and co-authored the UD release papers; his earlier work on parser adaptation between related languages anticipated direct transfer.
• Jan Hajič (Charles University and the Czech Academy of Sciences) coordinated the CoNLL-2009 joint syntactic–semantic task and is a core contributor to UD.
• Joakim Nivre (Uppsala University and RISE) is the leading figure of data-driven dependency parsing: he organised CoNLL 2007 and is first author of the UD v1 and v2 papers.
• Mikel Artetxe (University of the Basque Country) wrote several of the most influential cross-lingual papers here, from VecMap and LASER to work on translation artefacts and the XGLM models.
• Orhan Firat introduced multi-way translation with shared attention and later co-led Google's work on massively multilingual translation.
• Holger Schwenk developed joint multilingual sentence representations and LASER, and co-created MLQA and the MLDoc classification corpus.
• Iryna Gurevych (TU Darmstadt) co-authored multilingual knowledge distillation, MAD-X and the tokenizer study, linking sentence embeddings and modular adaptation.
• Ivan Vulić contributed to XCOPA, MAD-X and the influential critique of zero-shot transfer in From Zero to Hero.
• Graham Neubig, long at Carnegie Mellon University, worked on pre-trained embeddings for low-resource translation, rapid adaptation of translation to new languages and XTREME.
• Melvin Johnson co-authored Google's large-scale multilingual translation studies and the XTREME benchmarks.
• Daniel Cer helped run the multilingual semantic textual similarity tasks and co-developed LaBSE and the multilingual Universal Sentence Encoder.
• Eneko Agirre (University of the Basque Country) co-organised the SemEval-2014 and 2016 similarity tasks and co-authored VecMap.
• Chris Dyer co-authored the CCA-based multilingual word vectors, massively multilingual word embeddings, Many Languages, One Parser and cross-lingual metaphor detection.
• Ryan McDonald built one of the leading two-stage multilingual dependency parsers of CoNLL-X and co-authored cross-lingual word clusters for direct transfer.
• Thien Huu Nguyen (University of Oregon) leads recent work on large models beyond English: the ChatGPT multilingual evaluation, Okapi, CulturaX and the Trankit toolkit.
• Massimo Poesio (Queen Mary University of London) represents the multilingual coreference line, from the MATE scheme to the BART anaphora system and cross-lingual zero pronoun resolution.

Other names recur in particular strands: Milan Straka and Martin Popel in the UD shared tasks and multilingual coreference, Filip Ginter and Sampo Pyysalo of the University of Turku in UD and Finnish BERT, Naman Goyal in large multilingual pretraining, Barry Haddow in low-resource translation and EuroLLM, and Francis M. Tyers in Common Voice and UD.

Institutions and places

The research is concentrated in a handful of strong groups. Carnegie Mellon University, the University of Washington and the University of Edinburgh lead by the current affiliations of authors here, with Charles University in Prague as the centre of Universal Dependencies and multilingual coreference work. Industrial labs — Google, Google DeepMind, Microsoft and Amazon — produced much of the large-scale translation and pretraining research. In Europe, the University of Copenhagen and IT University of Copenhagen, Uppsala University, the University of the Basque Country and the French CNRS and Inria are prominent. Johns Hopkins University, Stanford University, MIT, MBZUAI, the Chinese University of Hong Kong and the University of Hong Kong complete the list.

By country, the United States leads, followed by the United Kingdom, China and Germany.

The Language Resources and Evaluation Conference (LREC) is the single largest venue here, which reflects how much of this topic is about building corpora, treebanks and benchmarks. EMNLP, ACL, the Transactions of the ACL and the CoNLL shared-task proceedings carry most of the methodological work, and workshops on under-resourced and endangered languages host a growing share of recent papers.

Open questions

• Who is left out. Benchmarks such as AmericasNLI and SIB-200 show that models still perform poorly on indigenous and many African and Asian languages; handling languages unseen by mBERT remains hard, especially for unfamiliar scripts.
• Transfer versus interference. How should capacity be shared among hundreds of languages, and can modular designs remove the curse of multilinguality rather than postpone it?
• Why transfer works. Analysis papers disagree about the roles of shared vocabulary, parameter sharing, word order and typological similarity. Studies of latent English-like processing and language-specific neurons open the question again for large models.
• Tokenisation. Subword vocabularies fitted mostly to large languages split other languages into many more tokens, raising cost and lowering quality. Byte- and character-level models are one proposed answer.
• Data quality. Web-mined corpora for low-resource languages are often mislabelled or noisy; auditing and documenting data is still not routine.
• Evaluation. Translated test sets carry artefacts and cultural bias toward English-speaking contexts, as Global MMLU and work on cultural values in language models emphasise. Native data collection is expensive.
• Safety across languages. Safeguards trained mainly in English can be bypassed in other languages, as shown by work on multilingual jailbreaks.

Neighbouring topics

The largest overlap is with Language models & pretraining: multilingual encoders and generative models are pretrained language models, and much of the recent work here is about how they behave across languages. Machine translation shares architectures, parallel corpora and the idea of a common multilingual representation. Coreference resolution contributes multilingual shared tasks and harmonised corpora, and Evaluation & benchmarks contains the cross-lingual leaderboards.

Several language-specific topics intersect: Semitic languages and Hebrew & Aramaic, where AraBERT and other dedicated models were compared with multilingual ones, and Low-resource languages, for which cross-lingual transfer is a primary strategy. Task-based neighbours include Sentiment & emotion, Named entity recognition, Question answering, Dependency parsing and Universal Dependencies, Discourse analysis, Dialogue systems and Text generation. Speech & spoken language and Vision & language extend multilinguality beyond text, while Idioms & multiword expressions, Reasoning, Tokenization & segmentation and Bias, fairness & safety raise questions that look different in each language.

Most of the papers stand on the Linguistics & languages shelf, with large shares on Natural language processing and Machine translation, and a few on AI & machine learning and Law.

Key works in this library

• How Multilingual is Multilingual BERT? — probes zero-shot transfer of mBERT across scripts and word orders.
• Beto, Bentz, Becas — mBERT as a general zero-shot baseline on five tasks.
• Are All Languages Created Equal in Multilingual BERT? — shows how quality falls with Wikipedia size.
• Universal Dependencies v1 — the design of harmonised cross-language syntax annotation.
• CoNLL-X Shared Task — the first large multilingual parsing comparison.
• CoNLL 2017 Shared Task — parsing raw text into UD for dozens of languages.
• Inducing Multilingual Text Analysis Tools — annotation projection through aligned bitext.
• Inducing Crosslingual Distributed Representations of Words — early bilingual neural embeddings for document classification.
• Massively Multilingual Word Embeddings — one vector space for dozens of languages from dictionaries.
• VecMap unsupervised mapping — bilingual word mappings with no bilingual data.
• LASER — a language-agnostic sentence encoder from multilingual translation.
• LaBSE — BERT-based sentence vectors for bitext retrieval.
• Multilingual knowledge distillation — extending English sentence embeddings to new languages.
• Cross-lingual transferability of monolingual representations — transfer without a shared vocabulary.
• MAD-X — adapters for languages unseen in pretraining.
• How Good is Your Tokenizer? — tokenisation as a cause of multilingual weakness.
• From Zero to Hero — limits of zero-shot transfer and the value of a few target examples.
• Multi-way multilingual NMT — shared attention across many language pairs.
• Improving massively multilingual NMT and zero-shot translation — capacity, off-target output and the OPUS-100 corpus.
• XTREME — a multitask benchmark for cross-lingual generalisation.
• TyDi QA — native questions in typologically diverse languages.
• XCOPA — causal commonsense with typologically sampled languages.
• Quality at a Glance — a manual audit of multilingual web corpora.
• XLS-R — cross-lingual speech pretraining at scale.
• Crosslingual Generalization through Multitask Finetuning — BLOOMZ, mT0 and the xP3 mixture.
• Multilingual Machine Translation with Large Language Models — how prompted LLMs compare with dedicated translation systems.