📄 Exploiting Lexical Resources for Disambiguating CJK and Arabic Orthographic Variants

Summary

This position paper examines why orthographic variation in Chinese, Japanese, Korean and Arabic makes retrieval, NER, MT, segmentation and transcription difficult. It argues that statistical or algorithmic methods are not enough and that large curated lexical databases should do most of the work. For Chinese, it compares three increasingly sophisticated levels of Simplified/Traditional (C2C) conversion: code-point mapping, word-level orthographic mapping and meaning-level lexemic mapping. It also covers normalization of Traditional character-form variants. For Japanese, it catalogues okurigana, cross-script and kana variants and shows that mapping tables can reduce them to a single normalized headword. Korean is covered only briefly through loanword and foreign-name spellings. For Arabic, it explains the ambiguity caused by unwritten short vowels and hamza rules. It also shows how foreign vowel sequences, such as Japanese morae, lead to competing spellings. The paper reports no controlled experiment. Its evidence is illustrative tables and web hit counts. The main lesson is that some variant pairs share no characters at all, so only a lexicon can link them.

Key points

• C2C conversion is described on three levels. Code conversion fails on one-to-many mappings (e.g., SC 干 maps to four TC forms). Orthographic conversion works on segmented words. Lexemic conversion maps different words with the same meaning (信息 → 資訊), and the correct TC form can depend on locale (Taiwan vs. Hong Kong).
• Osama bin Laden has more than 10 Chinese renderings. A single Japanese proverb produces 24 spelling permutations that actually occur on the web.
• A normalization table for 空き缶 'empty can' maps 11 surface forms to one headword. The paper says portals such as Yahoo and Amazon Japan use tables like this to improve recall and segmentation.
• Arabic transcriptions of 福井 /fu-ku-i/ show Google counts of 468, 9, 1950 and 335. The most phonologically accurate spelling is the rarest.
• Alexandria has about a dozen Arabic spellings (2,930,000 / 690,000 / 89,200 hits for the top three). The paper separates systematic "variants" (e.g., omitted hamza) from frequent "errors" (e.g., 'alif maqSuura used instead of yaa').

Interesting ideas

• I1. Treat conversion as a ladder: map single symbols first, then whole words, then meaning-equivalent lexemes, and move up a level whenever the lower one is ambiguous.
• I2. Only merge variants into a canonical form when they are fully interchangeable, and keep partially interchangeable pairs (different meanings in some senses) as separate entries.
• I3. Normalization only works if every component uses the same table: dictionaries, indexes, source documents and queries.
• I4. Keep "legitimate variant", "systematic error" and "linguistically standard form" as separate labels, since the correct form is often not the most frequent one.
• I5. Record variant-group membership as well as the replacement, so forms with no shared characters still link to one headword.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): A lexemic layer could link dialect pairs like lorry/truck to one concept node, so the world model treats them as the same entity type.
• I2 (tokens): The lemmatizer could apply spelling normalization (colour→color) automatically but leave sense-dependent pairs for the tagger or a later stage to resolve.
• I3 (answer): Normalize questions and story text with the same table so a reader cannot miss a match because of a spelling difference.
• I4 (learn): Self-induced repair rules could tag each fix as a variant or an error, making the corrections explainable and preventing frequency alone from deciding the canonical form.
• I5 (world): Entity aliases from Wikidata could be stored as explicit variant groups, letting coreference connect name spellings that share no surface overlap.