📄 ANETAC: Arabic Named Entity Transliteration and Classification Dataset

Summary

Research on Arabic named entity transliteration has been held back by a lack of training data. The only freely available English-Arabic resource the authors found has fewer than 13k pairs. This paper releases ANETAC, a dataset of 79,924 English-Arabic named entity pairs. Each pair is labelled as Person, Location, or Organization. The data was mined automatically from about 35 million parallel sentences on OPUS.

The extraction pipeline works in four steps:

• The Stanford NER tags entities on the English side.
• Multi-word names are split into single words.
• Frequent Arabic words are removed from the aligned sentence, so the remaining rare words serve as candidates.
• A rough polyglot transliteration of the English word is compared against each candidate using three string-similarity features. The best candidate is kept only if its score clears a confidence threshold.

The dataset ships with fixed train, dev, and test splits. Character-level attention encoder-decoder baselines show that English-to-Arabic is largely solved (WER 5.40), while Arabic-to-English remains hard (WER 65.16). The resource therefore serves both as a benchmark and as training material for transliteration, NE classification, OOV handling in MT, and cross-lingual retrieval.

Key points

• Dataset size and classes: ANETAC contains 79,924 triplets of English entity, Arabic transliteration, and class: 61,662 Person, 12,679 Location, and 5,583 Organization.
• Splits: 75,898 train, 1,004 dev, and 3,013 test instances. The dataset is hosted on GitHub.
• Source corpora: United Nation (10.6M sentences), Open Subtitles (24.4M), News Commentary (0.2M), and IWSLT2016 (0.2M), about 35.4M sentences in total.
• Candidate scoring: the top 40,000 most frequent Arabic words are filtered out first. Each remaining candidate is scored by averaging three features against the polyglot transliteration: shared character count, longest common character sequence, and a length-difference penalty.
• Baselines: character-level attention encoder-decoder models. English-to-Arabic reaches WER 5.40 / CER 0.95; Arabic-to-English reaches WER 65.16 / CER 16.35.

Interesting ideas

• I1. Removing a language's most frequent words from an aligned sentence leaves a small set of rare tokens, which turns a hard alignment problem into cheap candidate selection, useful wherever names or terms must be matched across noisy pairs. — For Mova (world): when building the entity inventory from parallel or paraphrase texts, frequency filtering could narrow down which tokens are likely names before linking them to Wikidata entries.
• I2. A weak, approximate transliterator need not be correct, because it only serves as a reference point for scoring real corpus words; a cheap imperfect witness thus bootstraps a cleaner resource. — For Mova (learn): a rough heuristic could play the same role for silver data, ranking outputs of stronger witness parsers rather than producing labels itself.
• I3. The candidate score is a transparent average of a few interpretable string features plus a confidence cutoff, so every accepted pair can be explained and borderline cases simply dropped. — For Mova (learn): self-induced repair rules or silver-tree selection could use the same pattern of a few named, inspectable features and an abstain threshold, keeping each acceptance decision explainable.
• I4. Multi-word entities are split into single words because each part can be handled without context, which multiplies training pairs and simplifies the mapping. — For Mova (back): when translating through the meta-world, proper names could be rendered token by token from a name table instead of through the full generation path.
• I5. Reporting both WER and CER per direction exposes asymmetry and near-misses that a single word-level metric would hide (65% word errors yet only 16% character errors). — For Mova (learn): evaluation of lemmatization or generation could pair exact-match with character-level error to separate total failures from small slips.