📄 Stemming Tigrinya Words for Information Retrieval

Summary

Tigrinya is a Semitic language of Eritrea and Ethiopia with very few computational resources. Its words carry many prefixes, suffixes and internal changes, so a plain search index treats one word as many terms. The authors built a stemmer as part of a planned Tigrinya search engine. They crawled Tigrinya web pages from news, religious, political, sports and other sites, then used the resulting word lexicon to compile affix lists, a stop-word list and a stem dictionary.

The stemmer works as a pipeline:

• strip apostrophe-marked clitics and punctuation;
• unify letters that sound the same but are written differently;
• skip stop words;
• romanize the Ethiopic syllables;
• remove the longest matching affixes iteratively;
• collapse doubled consonants;
• check the result against the stem dictionary, using edit distance and corpus frequency as fallbacks.

Tested on words from a newspaper and a Bible, the stemmer averaged 89.3% accuracy. This beats an earlier rule-only Tigrinya stemmer and gives a usable building block for Tigrinya retrieval.

Key points

• The authors crawled a web corpus with a crawler built for Tigrinya content and derived a lexicon of 690,000 unique words. The corpus is released via Eri-NLP (eri-nlp.com).
• They manually compiled 153 prefixes and 204 suffixes. Affix removal picks the longest match but respects a minimum stem length; words need at least three consonants to be stripped further.
• Ethiopic script is romanized with SERA conventions (with a few exceptions) before stripping. This matters because suffix vowels are fused into syllable glyphs.
• Dictionary verification works in three steps:
  • accept an exact match;
  • otherwise collect dictionary stems contained in the stripped string and drop those beyond a Levenshtein cutoff;
  • choose the closest remaining stem, breaking ties by corpus frequency.
• Implemented in C#, the stemmer scored 89.92% on 1,200 Hadas Eritra words and 88.6% on 1,300 Bible words, averaging 89.3%. A prior rule-based Tigrinya stemmer reported 86%.

Interesting ideas

• I1. Transliterating to an explicit consonant+vowel representation before stripping lets morpheme boundaries that hide inside a glyph become visible, so suffixes can be cut cleanly.
• I2. The pipeline proposes a stem with cheap rules, then accepts it only if a lexicon confirms it. Otherwise it picks the nearest contained dictionary entry under a distance cap, with frequency as the tie-breaker.
• I3. Spelling variants from regional conventions (Eritrean vs Ethiopian letter series) are merged before any other analysis, so they do not split counts or matches.
• I4. A minimum-length guard on the remaining stem is a simple, transparent brake against over-stemming short words.
• I5. The stemmer was evaluated separately on two very different genres (news vs scripture) rather than one pooled sample, which shows how stable the method is across domains.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Run suffix analysis on a decomposed form where spelling alternations are made explicit (e.g. "-ies" ↔ "y+s"), so the boundary is not guessed on the raw surface string.
• I2 (tokens): The lemmatizer could follow the same "rule proposes, lexicon disposes" pattern for unknown words, logging which branch fired so each lemma choice stays explainable.
• I3 (tokens): Normalize British/American spellings and similar variants ahead of lemma lookup, keeping the original form for generation.
• I4 (tokens): Unknown-word suffix stripping in the en crate could refuse cuts that leave too short a base, preventing errors like turning "news" into "new".
• I5 (learn): Report per-genre scores (school texts, fables, news) for each component, so a gain in one register is not hiding a loss in another.