Elamur Research⌕
☰
home / papers / 2026 / Rebelòt: Datasets and Token-Level Language Identification for Lombard-Italian-English Code-Mixing

Rebelòt: Datasets and Token-Level Language Identification for Lombard-Italian-English Code-Mixing

Summary

Lombard is an endangered Gallo-Italic variety used alongside Standard Italian. Speakers often mix it with Italian in daily speech and with English online, and it has no unified spelling. These factors make standard NLP tools unreliable on Lombard text, and the authors note that no earlier computational work had studied Lombard code-mixing. They release Rebelòt, a corpus spanning several domains in which every token is labelled as Lombard, Italian or English. On this data they build a word-level language identification (LID) system in three variants. Each variant fine-tunes a pre-trained encoder on a mix of real corpus sentences and synthetically generated code-mixed text. The best variant tags individual tokens with accuracy above 0.99. At the sentence level, it clearly outperforms common off-the-shelf identifiers. This matters for low-resource and endangered languages, where mixed text is common. Per-token language labels are useful for filtering corpora, preparing machine translation data such as subtitles, and choosing the right pronunciation rules in text-to-speech.

Key points

  • Resources: the Rebelòt dataset and the three LID model variants are publicly released at github.com/edoardosignoroni/rebelot.
  • Training data: the models combine authentic corpus data with synthetic code-mixed sentences. The best variant exceeds 0.99 token-level accuracy and beats FastText and GlotLID in sentence-level comparison.
  • Language status: Lombard has about 3.5 million speakers in Lombardy, eastern Piedmont, Ticino and Graubünden. UNESCO classifies it as "Definitely endangered", and on EGIDS it falls between 6b and moribund.
  • Declining use (ISTAT): in 2024, 4.2% of Lombardy residents spoke only the local variety at home and 18.1% used it together with Italian. In 2015 these figures were 5.6% and 26.1%.
  • Gaps in the source text: the excerpt provided did not include the methodology and results sections. Corpus size, domain breakdown, the exact encoder model and the full metric tables therefore still need to be checked against the full paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4yids37agyxu

@inproceedings{signoroni2026rebelot,
  title = {{Rebelòt: Datasets and Token-Level Language Identification for Lombard-Italian-English Code-Mixing}},
  author = {Signoroni, Edoardo and Bednaříková, Emma and Rychly, Pavel},
  booktitle = {Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages},
  year = {2026},
  doi = {10.63317/4yids37agyxu},
  url = {https://doi.org/10.63317/4yids37agyxu}
}

References

43 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Giovanna Alfonzetti. 2015. Age-related variation in code-switching between italian and the sicilian dialect. ATHENS JOURNAL OF PHILOLOGY, 2:21–34. find
  2. Giulia Andreoli. 2022. In italiàn t’el dìsi. italiano e dialetto a poggiridenti (so). Master’s thesis, Università degli Studi di Pavia. find
  3. Marino Anesa and Mario Rondi. 1981. Mondo Popolare in Lombardia - Fiabe Bergamasche, 1st edition. Silvana Editoriale, Cinisello Balsamo. find
  4. Emma Bednaříková and Pavel Rychlý. 2025. Evaluating training data construction strategies for token-level language identification. In Recent Advances in Slavonic Natural Language Processing (RASLAN 2025), pages 45–53. Tribun EU. find
  5. Gaetano Berruto. 1987. Lingua, dialetto, diglossia, dilalìa. In Johannes Holtus, Günter e Kramer, editor, Romania et Slavia adriatica. Festschrift für Zarko Muljačić, 1st edition, pages 57–81. Buske, Hamburg. find
  6. Gaetano Berruto. 1997. Code-switching and code-mixing. In Martin Maiden and Mair Parry, editors, The Dialects of Italy, 1st edition, chapter 46, pages 394–400. Routledge, London. find
  7. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, Inc. find
  8. Giovanni Bonfadini. 2010. lombardi, dialetti. In Treccani Eds., editor, Enciclopedia dell’italiano. Treccani. find
  9. Massimo Cerruti. 2018. Code-switching in italo-romance: a variationist study of convergence in bilingual speech. Lingue e linguaggio, Rivista semestrale, (1/2018):87–106. find
  10. Paolo Coluzzi, Lissander Brasca, and Simona Scuri. 2021. Revitalizing contested languages: The case of lombard. In Marco Tamburelli and Mauro Tosco, editors, Contested Languages: The hidden multilingualism of Europe, chapter 9, pages 163–182. John Benjamins, Amsterdam. find
  11. Paolo Coluzzi, Lissander Brasca, Marco Trizzino, and Simona Scuri. 2018. Language planning for italian regional languages: the case of lombard and sicilian. In Dieter Stern, Motoki Nomachi, and Bojan Belić, editors, Linguistic Regionalism in Eastern Europe and Beyond: Minority, Regional and Literary… find
  12. Silvia Dal Negro and Simone Ciccolone. 2018. Il parlato bilingue: Italiano e tedesco a contatto in un corpus sudtirolese. In Lingua parlata: Un confronto fra l’italiano e alcune lingue europee, page 23. Lang, Berlin. find
  13. John Lafferty, Andrew McCallum, Fernando Pereira, et al. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Icml, volume 1, page 3. Williamstown, MA. find
  14. Anik Dey and Pascale Fung. 2014. A hindi-english code-switching corpus. In LREC, pages 2410– 2413. find
  15. Caroline Lavecchia, Kamel Smaïli, and David Langlois. 2007. Building Parallel Corpora from Movies. In The 4th International Workshop on Natural Language Processing and Cognitive Science - NLPCS 2007, Funchal, Madeira, Portugal. find
  16. Ilaria Fiorentini. 2017. Segnali di contatto. Italiano e ladino nelle valli del Trentino-Alto Adige. Fran-coAngeli. find
  17. Federica Frighetto. 2025. ‘mi no go parole. . . o forse si’ - code mixing tra italiano standard e dialetto veneto nei social media. Master’s thesis, Università "Ca’ Foscari" - Venezia. find
  18. John J. Gumperz. 1977. The sociolinguistic significance of conversational code-switching. RELC Journal, 8(2):1–34. find
  19. ISTAT. 2017. L’uso della lingua italiana, dei dialetti e di altre lingue in italia. https://www.istat.it/it/archivio/207961. Accessed: 2025-08-07. ↗
  20. ISTAT. 2026. L’uso della lingua italiana, dei dialetti e delle lingue straniere - anno 2024. https://www.istat.it/comunicato-stampa/luso-della-lingua-italiana-dei-dialetti-e-delle-lingue-straniere-anno-2024/. Accessed: 2026-02-17. ↗
  21. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2019. Automatic language identification in texts: A survey. Journal of Artificial Intelligence Research, 65:675–782. find our card
  22. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, and Tomas Mikolov. 2017a. Fasttext.zip: Compressing text classification models. find our card
  23. M. Paul Lewis and Gary F. Simons. 2010. Assessing endangerment: Expanding fishmans’s gids. find
  24. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923– 929, Portorož, Slovenia. European Language Resources Association (E… find our card
  25. Michele Loporcaro. 2009. Profilo Linguistico dei Dialetti Italiani, 1st edition. Manuali Laterza. Editori Laterza, Bari. find
  26. Martin Maiden and Mair Perry. 1997. The Dialects of Italy. Routledge. find
  27. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. arXiv preprint arXiv:2509.06888. find
  28. Christopher Moseley. 2010. Atlas of the World’s Languages in Danger. UNESCO Publishing. find
  29. Christopher Moseley and Alexandre Nicholas. 2010. Atlas of the World’s Languages in Danger, 3rd edition, volume 19 of Memory of Peoples. UN-ESCO, Paris. find
  30. Pieter Muysken. 2000. Bilingual Speech: A Typology of Code-mixing. Cambridge University Press. find
  31. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017b. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Associati… find
  32. Ravindra Nayak and Raviraj Joshi. 2022. L3Cube-HingCorpus and HingBERT: A code mixed Hindi-English dataset and BERT language models. In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference, pages 7–12, Marseille, France. European Language Resources Associ… find
  33. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics. find our card
  34. Doreen Osmelak and Shuly Wintner. 2023. The denglisch corpus of German-English code-switching. In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 42–51, Dubrovnik, Croatia. Association for Computational Linguistics. find
  35. Giulia Paganessi. 2017. Brazilian Bergamasch: an Italian language spoken in Botuverá (Santa Catarina, Brazil). Leiden University. find
  36. Alan Ramponi. 2024. Language varieties of Italy: Technology challenges and opportunities. Transactions of the Association for Computational Linguistics, 12:19–38. find
  37. Caroline Sabty, Islam Mesabah, Özlem Çetinoğlu, and Slim Abdennadher. 2021. Language identification of intra-word code-switching for arabic–english. Array, 12:100104. find
  38. Sebastin Santy, Anirudh Srinivasan, and Monojit Choudhury. 2021. BERTologiCoMix: How does code-mixing interact with multilingual BERT? In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 111–121, Kyiv, Ukraine. Association for Computational Linguistics. find
  39. Edoardo Signoroni and Pavel Rychlý. 2026. Lom-bardographia: Automatic classification of lombard orthography variants. find
  40. Igor Sterner and Simone Teufel. 2023. TongueSwitcher: Fine-grained identification of German-English code-switching. In Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code-Switching, pages 1–13, Singapore. Association for Computational Linguistics. find
  41. Ronald Wardhaugh. 2006. An Introduction to Sociolinguistics. Blackwell Publishing. find
  42. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory e… find our card
  43. Yuan Zhang, Jason Riesa, Daniel Gillick, Anton Bakalov, Jason Baldridge, and David Weiss. 2018. A fast, compact, accurate model for language identification of codemixed text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 328–337, Brussels, Belgium.… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/signoroni-2026-rebelot-datasets-token-level-language/