📄 Bangor University at WojoodNER 2024: Advancing Arabic Named Entity Recognition with CAMeLBERT-Mix

Summary

This paper reports Bangor University's entry to Subtask-1 of the WojoodNER 2024 shared task: closed-track, flat, fine-grained Arabic NER. Arabic is hard for NER because it has no capitalization, rich morphology and many dialects. The task's Wojood-Fine corpus adds many entity subtypes. The authors fine-tuned the pre-trained CAMeL-Lab model bert-base-arabic-camelbert-mix with a single linear layer on top. That layer assigns each token one of 51 entity types/subtypes or the non-entity 'O' tag, using BIO-style tagging. Subword tokens are aligned to word labels. Training used cross-entropy loss and AdamW, and the checkpoint with the lowest validation loss was kept.

On the blind test set the system scored 86.45% F1, against 95.87% on training data and 89.16% on validation. This places it below the organizers' baseline (89) and the top team (91). The work is mainly useful as a plain, reproducible fine-tuning reference point. Its train–validation gap shows how much a bare linear head can overfit on sparse fine-grained classes.

Key points

• Model: bert-base-arabic-camelbert-mix (Inoue et al., 2021), fine-tuned on Wojood-Fine (Liqreina et al., 2023) with one linear classification layer over 51 entity labels plus 'O'.

• Data splits: 70% train, 10% validation, 20% test. Test labels were withheld for blind evaluation.

• Hyperparameters: batch size 32, up to 50 epochs, AdamW, learning rate 1e-4, StepLR scheduler, cross-entropy loss. Early stopping halted training after 10 epochs without validation-loss improvement.

• Results by split:

  Split	Precision	Recall	F1
  Training	95.19%	96.57%	95.87%
  Validation	87.86%	90.49%	89.16%
  Testing	88.06%	84.90%	86.45%

• Leaderboard: mucAI led Subtask-1 with F1 91, and the baseline scored 89. Bangor's 86 was the lowest of the listed entries.

Interesting ideas

• I1. A single linear layer mapping contextual subword vectors directly to a large fine-grained label set gives a minimal, transparent tagging head that is easy to inspect and compare against richer heads.
• I2. Wordpiece tokens are aligned back to word-level labels so the subword encoder can be trained against word-level gold annotation.
• I3. Precision, recall and F1 are reported separately for train, validation and test, so the size of the generalization gap becomes a visible diagnostic rather than a hidden number.
• I4. Placing a system next to the organizers' baseline reveals that a plain fine-tune fell short, which an absolute score alone would hide.
• I5. The authors point to data sparsity in fine-grained classes as the likely source of weakness, which argues for checking per-class coverage before trusting aggregate scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A per-token linear classifier over encoder states could serve as the explainable floor for fine-grained entity or POS tagging, with any heavier tagger required to beat it.
• I2 (tree): gparse needs the same word↔subword alignment discipline to project UD heads and labels from treebank words onto transformer pieces and back without drift.
• I3 (learn): Evaluation reports should always print the train/dev/test triple, so silver-data overfitting shows up before it contaminates later stages.
• I4 (learn): Every new model should be scored against a fixed simple baseline acting as a negative control, and a gain should count only when it clears that bar.
• I5 (world): When entity types are compiled from WordNet/Wikidata into many subtypes, rare subtypes should be flagged and evaluated separately rather than averaged away.