📄 amLite: Amharic Transliteration Using Key Map Dictionary

Summary

Amharic NLP work has mostly been done on ASCII romanizations of the Ge'ez script ("fidel"). Researchers and writing tools each adopted their own conventions, so the same word can appear in many spellings. This makes datasets hard to share and makes byte-level cost comparisons unreliable. amLite tries to fix this by turning romanized Amharic back into native Unicode script.

The author built a key map dictionary that links every ASCII combination he found in use to its fidel character. The sources were SERA, earlier transliterated texts, the Unicode standard and existing keyboard software. Conversion sorts the keys so that longer ones come first, with ties broken by ascending ASCII order, and then replaces matches greedily. The Python implementation was tested on three files and produced 97.7%, 99.7% and 98.4% correct words.

Almost all remaining errors have one cause. Greedy longest-match joins adjacent vowels, or a letter and a digit, into a single key that crosses a syllable boundary. The work matters because it offers a practical way to normalize scattered Amharic resources into a single canonical form.

Key points

• The key map is built from 76 ASCII symbols (letters, digits, punctuation). One fidel character can have many romanizations; for example, ኋ appears in about eight variant spellings.

• SERA (System for Ethiopic Representation in ASCII) is the backbone scheme. A competing scheme that uses non-ASCII symbols such as ä was rejected as impractical.

• The pipeline has these steps:

  • read the input and the dictionary;
  • normalize encodings;
  • sort keys by length, then by ASCII order;
  • decode the input;
  • replace by longest match;
  • encode the output as UTF-8.

• The three test sets were:

  • a 32,482-word list with one word per line;
  • a poem of about 276 lines marking the centenary of the Adwa victory;
  • an Injera recipe of about 31 lines.

  Errors in the word list were counted only over the first 1,000 words, because the same failure kept recurring.

• Six romanizations of one name ranged from 27 to 28 bytes in memory, while the native UTF-8 string took 40 bytes. The author uses this to argue that benchmarks run on romanized text are not comparable.

Interesting ideas

• I1. Collapse many observed variant spellings into a single many-to-one table keyed by surface form, so every downstream consumer sees one canonical representation.
• I2. Greedy longest-match with a deterministic tie-break is simple and predictable, but it breaks when a long key spans a real unit boundary (e.g., "meaza" read as "me"+"a"+"za" gets wrongly grabbed as "ea").
• I3. The paper separates errors already present in the input (misspellings that need a lexicon to fix) from errors introduced by the mapping itself, and counts only the latter against the system.
• I4. Explicit boundary markers in the input, such as writing "me-a-za", are proposed as a cheap way to resolve segmentation ambiguity without changing the algorithm.
• I5. Keeping the mapping as an open list of key-to-character pairs lets coverage grow by adding entries rather than editing code.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A similar variant table could normalize typographic quotes, spelling variants and odd Unicode before tagging, keeping the original form for explainability.
• I2 (tokens): When the tokenizer matches multiword expressions or contractions by longest match, add boundary-aware exceptions and targeted tests for cases where the long match crosses a word or morpheme edge.
• I3 (learn): Evaluation reports could tag each failure as source-text noise or pipeline-induced, so repair rules target only errors Mova actually caused.
• I4 (tokens): Silver or hand-made test data could carry optional boundary hints for known-ambiguous strings, which would let the tokenizer be checked against intended segmentations.
• I5 (world): Compiled knowledge tables (measures, value words, ontology aliases) can follow the same plain-data, append-only design so they stay inspectable and easy to extend.