📄 Application of corpus-based techniques to Amharic texts

Summary

The authors look for ways to speed up lexicon building for Amharic. Amharic is a Semitic language with rich root-and-pattern morphology and very few NLP tools. The work belongs to a project that adds Amharic to the transfer-based CAT2 machine translation system, which needs an English–Amharic transfer lexicon.

Because no robust Amharic parser or morphological analyser existed, they did not adapt the linguistically heavy AUTOTERM term extractor. They chose a mostly statistical approach instead. They took 720,000 words of newspaper text in SERA transliteration and scored contiguous word pairs with the log-likelihood ratio. The strongest pairs were merged into single units, and this was repeated until no significant associations remained. This produced names, compounds, multiword function expressions and non-compositional phrases suitable for the lexicon. Patterns seen in a small hand-collected idiom sample also let them harvest many more idioms. A pilot study grouped words by part of speech from morphological features. Supervised classification beat the majority-class baseline, while k-means clustering did not. Overall, cheap corpus statistics prove useful for a low-resource language, but full morphological analysis remains necessary.

Key points

• The corpus has 720,000 words of Amharic newspaper text. It is encoded in the SERA Latin transliteration, which, like the native script, does not mark gemination.
• Bigram association was measured with the log-likelihood ratio and applied iteratively to grow tri-grams. Top results include adis abäba ('Addis Ababa'), mkr bet ('Council'), ngd mkr bet ('Chamber of Commerce') and qalä mləls ('interview').
• A small test searched for discontinuous pairs within a 5-word window. It found the month…qän ('day') pattern, which follows from the Amharic date format.
• Three inflected variants of Addis Ababa appear among the top 20 candidates. Stripping common prefixes and suffixes helped only slightly, because much grammatical information is encoded by stem-internal changes.
• The part-of-speech pilot represented words by recurring prefixes, suffixes, CV templates and vowel patterns. Classification under 10-fold cross-validation beat the 54% majority baseline; k-means did not. Adjectives and adverbs were often mislabelled as nouns.

Interesting ideas

• I1. Strong word pairs are fused into one token and association is recomputed, so longer fixed expressions grow step by step without a parser.
• I2. A handful of manually gathered idioms showed formation patterns, such as repeated word sequences and words that often occur in idioms, which then served as filters to find many more.
• I3. Seeing several inflected forms of one name crowd the top list is a cheap signal of data sparsity. It also measures how much shallow affix stripping actually recovers.
• I4. Every unsupervised or supervised result is reported against the trivial most-frequent-class baseline, which exposed that clustering added nothing.
• I5. Searching for collocations with gaps inside a small window revealed a fixed date template rather than just adjacent pairs.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A similar loop over a large English corpus could propose multiword units such as "on the other hand" for the tokenizer and lemma lexicon, each with a logged score as its explanation.
• I2 (frames): Seed patterns could mark English candidates as possibly non-compositional, so the frame builder does not assign literal roles to idioms.
• I3 (learn): Checking whether top-ranked items collapse after lemmatization could serve as a diagnostic of whether normalization in a pipeline stage is doing real work.
• I4 (learn): Requiring a majority-class baseline next to each new tagger or classifier score acts as a simple negative control against illusory gains.
• I5 (world): Mining gapped co-occurrences could uncover recurring templates for dates, amounts and measures that map directly onto typed values in the world model.