📄 An Amharic Stemmer : Reducing Words to their Citation Forms

Summary

The paper builds a stemmer for Amharic, a Semitic language with rich morphology. It is designed for dictionary-based cross-language IR, where query words must map to the exact citation form listed in a machine readable dictionary (MRD). Full morphological analysis is seen as unnecessary for this goal, and precision matters most.

The authors wrote 65 hand-crafted rules covering affixes and allowed prefix/suffix combinations per word category. The rules generate every possible segmentation of a word, and each candidate stem is checked against three MRDs. When several candidates match, or none do, the system chooses using occurrence statistics from the 3.1M-word ENH news corpus. Stems that fail to match get a small repair, such as restoring a final vowel, and are tried again. Texts are first transliterated into SERA.

Accuracy is 76.9% on news from Walta Information Center and 60% on a classic novel. This shows a clear drop under stylistic shift. The work offers a practical recipe for a low-resource language with no public morphological analyzer.

Key points

• Rules: 65 morphological rules, from simple affixation to legal prefix–suffix combinations per word category.
• Resources:
  • The ENH corpus (about 3.1M words in roughly 10,000 articles) supplies word and affix statistics.
  • Three MRDs (French–Amharic, Amharic–French, English–Amharic) total about 83,000 entries but only 47,176 unique ones.
• Evaluation data: 1503 WIC words (1000 unique) and a 470-word passage of "Fikir Iske Meqabir" (300 unique).
• WIC results by case (76.9% overall):

• FIM results (60% overall): 68.8% for a single match, 54.8% for several matches, 42.1% for no match.

Interesting ideas

• I1. Over-generate all rule-licensed segmentations and let a lexicon of citation forms act as the verifier, so the rules can stay permissive while the dictionary supplies precision.
• I2. Use a tiered back-off: an exact dictionary match first, then a lightly repaired stem (e.g. restoring a dropped final vowel), and only then corpus statistics.
• I3. When nothing matches the dictionary, prefer the shortest stem that is attested as a standalone word in a large corpus, using attestation as cheap weak evidence.
• I4. Report accuracy separately for unambiguous, ambiguous and unmatched cases, which shows where errors really come from and what future work should target.
• I5. Test on an out-of-domain text (an older novel versus news) on purpose to measure robustness to style drift.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The en lemmatizer could propose candidate lemmas for unseen inflected or derived words via suffix rules and keep only those found in the lexicon, logging which rule produced each one.
• I2 (tokens): Each lemma decision could record its tier, making the morphology step explainable and making low-confidence lemmas easy to flag for downstream stages.
• I3 (tokens): This could guide splitting of unknown compounds or hyphenated forms by picking pieces that occur independently in a reference corpus.
• I4 (learn): Evaluation could be bucketed by how many witness analyses agree or how many lexicon matches exist, so regressions in the hard buckets are not hidden by easy cases.
• I5 (learn): Evaluations could routinely pair in-domain sets with stylistically distant ones, such as old fables versus school texts, as a shift check.