Elamur Research⌕
☰
home / papers / 2026 / Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

Summary

Piedmontese is an endangered Romance language of northwestern Italy. Most existing NLP data for it comes from Wikipedia, which follows a standardized spelling that ordinary speakers often do not use. To test how LLMs handle the language as people actually write it, the authors crowdsourced translations through an Italian-language online questionnaire. Volunteers rendered randomly chosen Flores+ sentences into Piedmontese using whatever spelling felt natural, and other participants then peer-reviewed those translations.

The final resource has 145 Italian–Piedmontese sentence pairs, manual word alignments and the raw annotations with sociolinguistic metadata. The authors used it to evaluate open-weight and closed LLMs on three things: tokenizer parity, topic classification with SIB-200 labels, and zero-shot machine translation.

Key findings:

  • Piedmontese is over-tokenized compared with Italian, French and English.
  • Classification performance comes close to that of the high-resource languages.
  • Translating out of Piedmontese works reasonably well, but generating Piedmontese remains weak, with spelling that is inconsistent across models.

The openly released data gives a benchmark of authentic, non-standard writing for a low-resource language variety.

Key points

  • Data collected: 200 annotations were gathered, and 145 of them are valid translations (68 from the Flores+ dev split, 77 from devtest). Peer reviews were used only to filter out missing or offensive entries. The dataset is on LINDAT (hdl.handle.net/11372/LRT-6086), and the code is in the GitHub repository CrowdsourcedPiedmontese.
  • Word alignment: The manual alignment contains 3003 spans, 2902 of which link one word to one word. The eflomal baseline beats SimAlign with XLM-RoBERTa (F1 .774 vs .589).
  • Models tested: Llama 3.3 70B, Gemma 3 27B, Qwen 3 30B, EuroLLM 9B, Tower Plus 9B, Gemini 2.5 Flash and GPT-4o-mini. French and English served as comparison languages alongside Italian.
  • Tokenizer parity: Piedmontese has the worst parity among the languages compared, and closed models showed no advantage over open ones. SentencePiece BPE and UnigramLM tokenizers trained on Glot500 data (32k vocabulary) that includes Piedmontese reduce the penalty.
  • Annotator survey: Most annotators were young and mainly used Italian. Only 11% believe speakers follow a standard when writing Piedmontese, while 42% disagree.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.vardial-1.6

@inproceedings{vico2026crowdsourcing,
  title = {{Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography}},
  author = {Vico, Gianluca and Libovický, Jindřich},
  booktitle = {Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2026},
  doi = {10.18653/v1/2026.vardial-1.6},
  url = {https://doi.org/10.18653/v1/2026.vardial-1.6}
}

References

7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the Eu… find our card
  2. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Com… find our card
  3. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904… find our card
  4. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 o… find
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  6. Robert Östling and Jörg Tiedemann. 2016. Efficient word alignment with Markov Chain Monte Carlo. Prague Bulletin of Mathematical Linguistics, 106:125–146. find
  7. A Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all – adapting pre-training data processing to every language. Preprint, arXiv:2506… find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vico-2026-crowdsourcing-piedmontese-test-llms-non/