📄 Dense Retrieval for Low Resource Languages -- the Case of Amharic Language

Summary

The paper asks whether modern neural retrievers can work for Amharic, a Semitic language with about 120 million speakers. Its rich inflection and derivation produce many surface forms per word, which hurts exact-match ranking, and annotated data and tools are scarce. The authors compare BM25 (Lemur Indri) against ColBERTv2 built on several Amharic masked language models. They also try SPLADE. They create a small training set, Train Amharic, and evaluate on two collections: 2AIRTC and the Amharic part of AfriCLIRMatrix, whose English test queries they machine-translated into Amharic. Untuned encoders give near-zero scores, whether English or Amharic. After fine-tuning on only about 150 examples, ColBERT becomes usable, and the best model reaches NDCG@10 of 0.704 on 2AIRTC. SPLADE yields nothing. The authors also point to three practical problems: indexing is much more expensive, the pretraining text may overlap with the test documents, and local compute is lacking. The work matters because it suggests a tiny, manually built training set may be enough to adapt dense retrieval to under-served languages.

Key points

• Data:
  • 2AIRTC has more than 10,000 documents and 240 curated queries with human relevance judgments.
  • Train Amharic has 152 queries and 152 documents.
  • AfriCLIRMatrix (Amharic side) has 15,458 documents and 1,500 test queries, translated with NLLB-200.
• Encoders come from rasyosef on Hugging Face: bert-medium-amharic, roberta-base-amharic and roberta-amharic-text-embedding-base. They were pretrained on about 290M tokens from Oscar, C4 and the Amharic Sentences Corpus.
• Scores without fine-tuning were near zero:
  • English BERT models gave NDCG@10 of 0.000–0.061.
  • Amharic-pretrained models gave 0.000–0.005.
• The best fine-tuned model, roberta-amharic-text-embedding-base, scored NDCG@10 of 0.704 on 2AIRTC but only 0.177 on AfriCLIRMatrix. DistilSPLADE v2 produced only null results.
• Timing compared with Indri:
  • ColBERTv2 indexing took about 29× (2AIRTC) and 46× (AfriCLIRMatrix) longer.
  • Query time was roughly the same (0.78× and 1.17×).

Interesting ideas

• I1. About 150 hand-made query–document pairs were enough to turn a useless encoder into a working retriever, well below the roughly 1000 examples previously reported. Small curated seed sets can therefore be worth more than waiting for big data. — For Mova (learn): Before scaling silver data, try adapting a weak component with a small hand-checked seed set and measure how far that alone carries it.
• I2. The paper separates the effect of architecture from the effect of extra fine-tuning. The best model wins because of its additional tuning, not because of its design. — For Mova (learn): When a parser or teacher tier improves, run ablations that isolate the training data or tuning step from the model change, so gains are credited correctly.
• I3. The authors warn that in low-resource settings the pretraining corpus may contain the very documents used for evaluation, a form of collection contamination. — For Mova (learn): Check that held-out test texts (fables, school tests) never appear in silver or teacher data, and treat such overlap as a negative control that must fail.
• I4. A test set in the target language can be built by machine-translating queries from an existing English benchmark (here with NLLB-200). — For Mova (answer): Translated variants of QA or word-problem sets could probe whether answering depends on the meaning graph rather than on English surface wording.
• I5. The paper reports one-time indexing cost separately from per-query cost. That split shows a heavy upfront investment can still give fast answers. — For Mova (world): Measure the compiled knowledge layer (ontology, verb classes) the same way, keeping build cost and lookup cost as separate numbers so expensive precompilation stays justified.