Elamur Research⌕
☰
home / papers / 2025 / Evaluating Pixel Language Models on Non-Standardized Languages

Evaluating Pixel Language Models on Non-Standardized Languages

Paper · 2025 · Alberto Muñoz-Ortiz, Verena Blaschke and Barbara Plank

Summary

Pretrained language models perform poorly on dialects, partly because subword tokenizers split non-standard spellings into meaningless fragments. The authors ask whether pixel-based models, which render text as images and read it in 16×16 patches, transfer better from a standard language to its dialects. Using German as the test case, they pretrain a PIXEL model from scratch on the DBMDZ corpus and release it. They compare it with the cased and uncased DBMDZ German BERT models, which were trained on the same data. All three are fine-tuned on Standard German for four tasks (detecting intents, classifying topics, parsing dependencies and tagging parts of speech), then evaluated zero-shot on Bavarian, Alemannic (Swiss German, Alsatian), Low Saxon and Turkish–German code-switched data.

On Standard German, pixel trails BERT on every task. On dialects, it wins on tagging, parsing and intent detection, with gains of up to about 26 points, but it loses on topic classification. The study suggests that tokenization-free input can make models more robust to unstandardized spelling, though the evidence so far covers only German.

Key points

  • The model uses PIXEL's design: a renderer that turns text into an RGB image split into 16×16 patches, and a 12-layer ViT encoder with 86M parameters. It is pretrained by reconstructing masked spans of patches. The released checkpoint is amunozo/pixel-base-german on Hugging Face.

  • Syntactic models were trained on the UD German GSD and HDT treebanks. They were tested on the following data:

    • MaiBaam (Bavarian)
    • LSDC (Low Saxon)
    • SAGT (Turkish–German)
    • UZH (Swiss German)
    • NOAH's Corpus and Alsatian BISAME (POS only)

    All results are averaged over five runs.

  • With HDT training, pixel's advantage on dialects grows:

    • Swiss German UZH POS accuracy: 54.9 vs 45.1 for bert-cased.
    • Bavarian LAS: 29.4 vs 23.2.
    • Alsatian POS accuracy: 48.2 vs 34.1.

  • For intent detection on xSID 0.5, pixel reaches 83.8 on Swiss German, against at most 62.4 for BERT. It also wins on the Bavarian test sets except MAS:de-ba.

  • bert-uncased does well on Standard German but much worse on dialects. The authors think bert-cased uses noun capitalization to make up for bad tokenization.

Interesting ideas

  • I1. Dialectal spelling can be framed as a kind of input perturbation and measured by training only on the standard variety and testing zero-shot on dialects; this cleanly isolates robustness to orthographic drift.
  • I2. Comparing cased and uncased models showed that capitalization acts as a strong fallback cue when word segmentation breaks down, which reveals what a model leans on under noise.
  • I3. Splitting LAS by dependency length showed that pixel's known weakness on long arcs vanished on dialects, a pattern the overall score hides.
  • I4. Per-POS accuracy showed that memorization favours PROPN for token models, while visual similarity within NUM and PUNCT helps pixel; tag-level breakdowns thus show which mechanism drives each gain.
  • I5. A continuous representation of word form, rather than a fixed vocabulary, lets spelling variants of a known word land near each other instead of falling apart into random pieces.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): add held-out non-standard or noisy-spelling English test sets as robustness controls, so that gains on clean treebanks are not mistaken for general parsing ability.
  • I2 (tokens): run ablations that strip casing or other surface cues from tagger features to see which cues carry the tagger on unknown words.
  • I3 (tree): report LAS per arc length for both the perceptron parser and gparse, and for converted UD dialects, to find where each parser actually fails.
  • I4 (tokens): break tagger errors down by UPOS on out-of-domain text to decide where lexicon lookup helps and where shape-based features for unknown words would help more.
  • I5 (tokens): add a character-shape similarity fallback that maps misspelled or variant forms to known lemmas, while keeping every mapping inspectable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Alberto Muñoz-Ortiz, Verena Blaschke and Barbara Plank. 2025. Evaluating Pixel Language Models on Non-Standardized Languages. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{munozortiz2025evaluating,
  title = {{Evaluating Pixel Language Models on Non-Standardized Languages}},
  author = {Alberto Muñoz-Ortiz, Verena Blaschke and Barbara Plank},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.427/}
}

References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Noëmi Aepli and Simon Clematide. 2018. Parsing Approaches for Swiss German. In Proceedings of the 3rd Swiss Text Analytics Conference (SwissText), Winterthur, Switzerland. ↗
  2. Noëmi Aepli, Çağrı Çöltekin, Rob Van Der Goot, Tommi Jauhiainen, Mourhaf Kazzaz, Nikola Ljubešić, Kai North, Barbara Plank, Yves Scherrer, and Marcos Zampieri. 2023. Findings of the VarDial evaluation campaign 2023. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 251–261, Dubrovnik, Croatia. Association for Computational Linguistics. ↗
  3. Noëmi Aepli and Rico Sennrich. 2022. Improving zero-shot cross-lingual transfer between closely related languages by injecting character-level noise. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4074–4083, Dublin, Ireland. Association for Computational Linguistics. ↗
  4. Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. 2021. The low-resource double bind: An empirical study of pruning for low-resource machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3316–3333, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  5. Gabriel Bernier-Colborne, Serge Leger, and Cyril Goutte. 2022. Transfer learning improves French cross-domain dialect identification: NRC @ VarDial 2022. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 109–118, Gyeongju, Republic of Korea. Association for Computational Linguistics. find
  6. Verena Blaschke, Barbara Kovačić, Siyao Peng, Hinrich Schütze, and Barbara Plank. 2024. MaiBaam: A multi-dialectal Bavarian Universal Dependency treebank. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10921–10938, Torino, Italia. ELRA and ICCL. find
  7. Verena Blaschke, Hinrich Schütze, and Barbara Plank. 2023. Does manipulating tokenization aid cross-lingual transfer? a study on POS tagging for non-standardized languages. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (Var-Dial 2023), pages 40–54, Dubrovnik, Croatia. Association for Computational Linguistics. find
  8. Emanuel Borges Völker, Maximilian Wendt, Felix Hennig, and Arne Köhn. 2019. HDT-UD: A very large Universal Dependencies treebank for German. In Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019), pages 46–57, Paris, France. Association for Computational Linguistics.
  9. Galo Castillo-lópez, Arij Riabi, and Djamé Seddah. 2023. Analyzing zero-shot transfer scenarios across Spanish variants for hate speech detection. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 1–13, Dubrovnik, Croatia. Association for Computational Linguistics. find
  10. Helen Christen. 2019. Alemannisch in der Schweiz. In Joachim Herrgen and Jürgen Erich Schmidt, editors, Sprache und Raum: Ein internationales Handbuch der Sprachvariation, volume 4, pages 246–279. De Gruyter Mouton, Berlin, Boston. find
  11. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Pelin Dogan-Schönberger, Julian Mäder, and Thomas Hofmann. 2021. Swissdial: Parallel multidialectal corpus of spoken swiss german. arXiv preprint arXiv:2103.11401. ↗
  14. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009. ↗
  15. Nora Hollenstein and Noëmi Aepli. 2014. Compilation of a Swiss German dialect corpus and its application to PoS tagging. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 85–94, Dublin, Ireland. Association for Computational Linguistics and Dublin City University. find
  16. Jonas Lotz, Elizabeth Salesky, Phillip Rust, and Desmond Elliott. 2023. Text rendering strategies for pixel language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10155–10172, Singapore. Association for Computational Linguistics. find
  17. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal Dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92–97, Sofia, Bulgaria. Association for Computational Linguistics.
  18. Ludwig Merkle. 1993. Bairische Grammatik, 5th edition. Heinrich Hugendubel Verlag, Munich. find
  19. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association. ↗
  20. Phillip Rust, Jonas F Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023. Language modelling with pixels. In The Eleventh International Conference on Learning Representations. ↗
  21. Elizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7235–7252, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  22. Rob van der Goot, Ibrahim Sharaf, Aizhan Imankulova, Ahmet Üstün, Marija Stepanović, Alan Ramponi, Siti Oryza Khairunnisa, Mamoru Komachi, and Barbara Plank. 2021. From masked language modeling to translation: Non-English auxiliary tasks improve zero-shot spoken language understanding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2479–2497, Online. Association for Computational Linguistics. find
  23. Peter Wiesinger. 1983. Die Einteilung der deutschen Dialekte. In Werner Besch, Ulrich Knoop, Wolfgang Putschke, and Herbert E. Wiegand, editors, Ergebnisse dialektologischer Beschreibungen: Areale Bereiche deutscher Dialekte im Überblick, pages 807–960. De Gruyter Mouton, Berlin, Boston. find
  24. Miriam Winkler, Virginija Juozapaityte, Rob van der Goot, and Barbara Plank. 2024. Slot and intent detection resources for Bavarian and Lithuanian: Assessing translations vs natural queries to digital assistants. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14898–14915, Torino, Italia. ELRA and ICCL. find
  25. Özlem Çetinoğlu and Çağrı Çöltekin. 2022. Two languages, one treebank: Building a Turkish–German code-switching treebank and its challenges. Language Resources and Evaluation, pages 1–35. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ortiz-2025-evaluating-pixel-language-models-non/