📄 OcWikiDialects: A Wikipedia Dataset With Rich Metadata for Occitan Dialect Identification

Summary

Occitan is a dialect continuum of southern France and nearby areas of Spain and Italy. Most NLP tools and standards such as ISO 639 treat it as a single language, and no public system existed for telling its varieties apart. The main obstacle is the lack of dialect-annotated text. The authors address this with OcWikiDialects, a corpus built from Occitan Wikipedia. Clean text comes from FineWiki, and the full-history XML dump supplies dialect tags, revision histories and contributor information, including self-declared proficiency, declared dialect and bot status. The text is split into paragraphs and sentences.

Combining this corpus with earlier dialect-labelled resources, the team trained n-gram SVM, FastText and fine-tuned BERT classifiers to distinguish up to eight varieties. They evaluated them on both in-domain and out-of-domain test sets. The fine-tuned oc-mBERT performed best. An ablation shows that adding the Wikipedia data clearly helps, above all for rare dialects. Accuracy still varies widely across varieties, so less-documented Occitan dialects need more resources.

Key points

• Size: OcWikiDialects contains 7,292 dialect-tagged articles, about 57k paragraphs, 290k sentences and 4M tokens.
• Labels: There are ten dialect labels. Provençal and Languedocian each cover roughly 2k articles, and Provençal alone exceeds 2M tokens. Auvergnat and Vivaroalpine are scarce but still better covered than in other datasets.
• Contributors: Only 35 of 1,249 active editors stated a dialect on their user page.
• Best result: Without text preprocessing, oc-mBERT reached a top macro F1 of 58.15 on the combined (CONCAT) test set.
• Ablation and errors: Removing the Wikipedia training data lowers macro F1 by as much as 23.48 points for oc-mBERT. Without it, the model never predicts Aranese, Auvergnat or Vivaroalpine. Rare classes suffer from low recall, while Languedocian tends to be over-predicted.