Elamur Research⌕
☰
home / papers / 2026 / “We Are (Language) Family”: Adapting Transformer models to related minority languages with linguistic data

“We Are (Language) Family”: Adapting Transformer models to related minority languages with linguistic data

Paper · 2026 · Miguel López-Otal and Jorge Gracia

Summary

Low-resource languages rarely get Transformer support because adapting a model usually needs large raw-text corpora. This paper asks whether linguist-made resources can stand in for, or supplement, such text when the source and target languages are closely related. The authors adapt two Spanish encoders, the monolingual roberta-base-bne and the multilingual mRoBERTa, to standard Aragonese, a Romance language of Northern Spain. Aragonese shares most of its syntax with Spanish but differs in vocabulary and morphology. They build an Aragonese–Spanish inflected lexicon that lists every word form with its part of speech, derived from Apertium data. They inject this knowledge with two adapter methods, Lauscher et al. (2020) and K-Adapter, and also reproduce the Gurgurov et al. (2024) recipe, which pairs knowledge injection with a MAD-X-style text adapter. On machine-translated Aragonese NLU benchmarks, the monolingual model edges past baselines, including a retrained MAD-X. It stays competitive even with linguistic data alone. The multilingual model does not benefit. The results suggest that adaptation with little or no corpus is possible within a language family.

Key points

  • Target language: Aragonese, spoken by fewer than 50,000 people. The paper notes it has no PoS taggers or Transformer models, and most prior NLP work has been machine translation, e.g. the Apertium module.
  • Main new resource: a prototype inflected lexicon for Aragonese–Spanish. It is released with training data, test data and evaluation scripts at github.com/sid-unizar/lrl-transformer-adaptation.
  • Evaluation uses silver-standard Aragonese NLU benchmarks machine-translated from Spanish (Baucells et al., 2025), including PAWS-X and COPA. On PAWS-X, linguistic data alone reached results comparable to text-based adaptation.
  • mRoBERTa underperformed its baselines, except for small COPA gains that may be unreliable given that dataset's small size. The authors suggest a cause: its pre-training on Catalan, Galician and French may already cover features Aragonese shares with those languages but not with Spanish.
  • Stated limitations:
    • Apertium-derived data was used for both training and evaluation, which may introduce bias.
    • The test sets are machine-translated, with manual revision planned.
    • COPA was used unconventionally for training.
    • Planned future work includes decoder models and NLG tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidad de ZaragozaZaragoza · Spain5 works · 5 citations🎓Instituto de Investigación en Ingeniería de AragónZaragoza · Spain2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Low-resource languages · 267 works

Cite

Cite as Miguel López-Otal and Jorge Gracia. 2026. “We Are (Language) Family”: Adapting Transformer models to related minority languages with linguistic data. Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.26

@inproceedings{lopezotal2026we,
  title = {{“We Are (Language) Family”: Adapting Transformer models to related minority languages with linguistic data}},
  author = {Miguel López-Otal and Jorge Gracia},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.26},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.26}
}

References

50 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a. Translation artifacts in cross-lingual transfer learning. arXiv preprint arXiv:2004.04721.
  2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4623–4637.
  3. Niyati Bafna, Emily Chang, Nathaniel Romney Robinson, David R. Mortensen, Kenton Murray, David Yarowsky, and Hale Sirin. 2025. DialUp! modeling the language continuum by adapting models to dialects and dialects to models. In Proceedings of the 63rd Annual Meeting of the Association for Computational… find
  4. Irene Baucells, Javier Aula-Blasco, Iria de Dios-Flores, Silvia Paniagua Suárez, Naiara Perez, Anna Salles, Susana Sotelo Docio, Júlia Falcão, José Javier Saiz, Robiert Sepúlveda-Torres, and 1 others. 2025. Iberobench: A benchmark for llm evaluation in iberian languages. In Proceedings of the 31st I…
  5. Sabri Boughorbel, Md Rizwan Parvez, and Majd Hawasly. 2024. Improving language models trained on translated data with continual pre-training and dictionary learning analysis. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 73–88, Bangkok, Thailand. Association for C…
  6. Junhao Chen, Peng Shu, Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Yifan Zhou, Zhengliang Liu, Lewis C Howe, and Tianming Liu. 2024. Queen: A large language model for quechua-english translation. Preprint, arXiv:2412.05184. — [arXiv:2412.05184](https://arxiv.org/abs/2412.05184) find
  7. Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. 2023. Language-family adapters for low-resource multilingual neural machine translation. In Proceedings of the Sixth Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2023), pages 59–72, Dubrovnik, Cro… find
  8. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2475–2485. find
  9. Ander Corral, Ixak Sarasua Antero, and Xabier Saralegi. 2025. Pipeline analysis for developing instruct LLMs in low-resource languages: A case study on Basque. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Languag… find
  10. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  11. Estudio de Filología Aragonesa. 2021. Gramática básica de l’aragonés. Prensas de la Universidad de Zaragoza. find
  12. Julen Etxaniz, Oscar Sainz, Naiara Miguel, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormazabal, Mikel Artetxe, and Aitor Soroa. 2024. Latxa: An open language model and evaluation suite for basque. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu… find
  13. Mikel L Forcada, Mireia Ginestí-Rosell, Jacob Nordfalk, Jim O’Regan, Sergio Ortiz-Rojas, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Gema Ramírez-Sánchez, and Francis M Tyers. 2011. Apertium: a free/open-source platform for rule-based machine translation. Machine translation, 25(2):127– 144. find
  14. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. 2024. Continual pre-training for cross-lingual llm adaptation: Enhancing japanese language capabilities. Preprint, arXiv:2404.17790.
  15. Pablo Gamallo, Pablo Rodríguez, Iria de Dios-Flores, Susana Sotelo, Silvia Paniagua, Daniel Bardanca, José Ramom Pichel, and Marcos Garcia. 2024. Open generative large language models for galician. arXiv preprint arXiv:2406.13893. — [arXiv:2406.13893](https://arxiv.org/abs/2406.13893) find
  16. Luke Gessler and Amir Zeldes. 2022. Microbert: Effective training of low-resource monolingual berts through parameter reduction and multitask learning. arXiv preprint arXiv:2212.12510. find
  17. Antoni Oliver González. 2024. Sistemas de traducción automática para el aragonés. Luenga & fablas: pub-licazión añal de rechiras, treballos e decumentazión arredol de l’aragonés ea suya literatura, (28):91– 100. find
  18. Raphaël Merx, Aso Mahmudi, Katrina Langford, Leo Alberto de Araujo, and Ekaterina Vylomova. 2024. Low-resource machine translation through retrieval-augmented llm prompting: A study on the mambai language. arXiv preprint arXiv:2404.04809.
  19. Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, and 1 others. 2025. Salamandra technical report. arXiv preprint arXiv:2502.08489. — [arXiv:2502.08489](https://arxiv.org/abs/2502.08489) find
  20. Marinela Parović, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2022. Bad-x: Bilingual adapters improve zero-shot cross-lingual transfer. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1791–179… find
  21. Daniil Gurgurov, Mareike Hartmann, and Simon Ostermann. 2024. Adapting multilingual llms to low-resource languages with knowledge graphs via adapters. arXiv preprint arXiv:2407.01406.
  22. Daniil Gurgurov, Ivan Vykopal, Josef van Genabith, and Simon Ostermann. 2025. Small models, big impact: Efficient corpus and graph-based adaptation of small multilingual language models for low-resource languages. arXiv preprint arXiv:2502.10140. find
  23. Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies, Joan Llop-Palao, Joaquin Silveira-Ocampo, Casimiro Pio Carrino, Aitor Gonzalez-Agirre, Carme Armentano-Oller, Carlos Rodriguez-Penagos, and Marta Villegas. 2021. Maria: Spanish language models. arXiv preprint arXiv:2107.07253. — [arXiv:2107.07253](https://arxiv.org/abs/2107.07253) find
  24. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR. find
  25. Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2023. A survey of knowledge enhanced pre-trained language models. Preprint, arXiv:2211.05994. find
  26. Andrew Johnson, Penny Karanasou, Judith Gaspers, and Dietrich Klakow. 2019. Cross-lingual transfer learning for Japanese named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… find
  27. Anne Lauscher, Olga Majewska, Leonardo FR Ribeiro, Iryna Gurevych, Nikolai Rozanov, and Goran Glavaš. 2020. Common sense or world knowledge? investigating adapter-based knowledge injection into pretrained transformers. arXiv preprint arXiv:2005.11787. find
  28. Miguel López-Otal, Jorge Gracia, Jordi Bernad, Carlos Bobed, Lucía Pitarch-Ballesteros, and Emma Anglés-Herrero. 2025. Linguistic interpretability of transformer-based language models: a systematic review. Preprint, arXiv:2504.08001. — [arXiv:2504.08001](https://arxiv.org/abs/2504.08001) find
  29. Renhao Pei, Yihong Liu, Peiqin Lin, François Yvon, and Hinrich Schuetze. 2025. Understanding in-context machine translation for low-resource languages: A case study on Manchu. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87… find
  30. Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Víctor M Sánchez-Cartagena, Miquel Esplà-Gomis, Aarón Galiano-Jiménez, Antoni Oliver, Claudi Aventín-Boya, Alejandro Pardos, Cristina Valdés, Jusèp Loís Sans Socasau, and 1 others. 2024. Expanding the flores+ multilingual benchmark with translations… find
  31. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume, pages 487–503…
  32. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,…
  33. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.00052.
  34. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association…
  35. Rita Ramos, Everlyn Asiko Chimoto, Maartje Ter Hoeve, and Natalie Schluter. 2025. GrammaMT: Improving machine translation with grammar-informed in-context learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29920– 29940,…
  36. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90–95. find
  37. Felipe Sánchez-Martínez, Juan Antonio Perez-Ortiz, Aaron Galiano Jimenez, and Antoni Oliver. 2024. Findings of the WMT 2024 shared task translation into low-resource languages of Spain: Blending rule-based and neural systems. In Proceedings of the Ninth Conference on Machine Translation, pages 684–6… find
  38. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  39. Lütfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schuetze, and Goran Glavaš. 2024. Kardeş-NLU: Transfer to low-resource languages with the help of a high-resource cousin – a benchmark and evaluation for Turkic languages. In Proceedings of the 18th Conference of the European Chapter of th… find
  40. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2023. A benchmark for learning to translate a new language from one grammar book. arXiv preprint arXiv:2309.16575. find
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  42. Ivan Vykopal, Simon Ostermann, and Marián Šimko. 2025. Soft language prompts for language transfer. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10294–10313. find
  43. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuan-Jing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405–1418.
  44. Jian Yang, Xinyu Hu, Gang Xiao, and Yulong Shen. 2024a. A survey of knowledge enhanced pre-trained language models. ACM Trans. Asian Low-Resour. Lang. Inf. Process. Just Accepted. find
  45. Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. 2024b. Give us the facts: Enhancing large language models with knowledge graphs for fact-aware language modeling. IEEE Trans. on Knowl. and Data Eng., 36(7):3091–3110. find
  46. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. Paws-x: A cross-lingual adversarial dataset for paraphrase identification. arXiv preprint arXiv:1908.11828.
  47. Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. 2019. A review of recurrent neural networks: Lstm cells and network architectures. Neural computation, 31(7):1235–1270. find
  48. Chen Zhang, Jiuheng Lin, Xiao Liu, Zekai Zhang, and Yansong Feng. 2025. Read it in two steps: Translating extremely low-resource languages with code-augmented grammar books. Preprint, arXiv:2506.01796. find
  49. Kexun Zhang, Yee Choi, Zhenqiao Song, Taiqi He, William Yang Wang, and Lei Li. 2024. Hire a linguist!: Learning endangered languages in llms with in-context linguistic descriptions. In Findings of the Association for Computational Linguistics ACL 2024, pages 15654–15669. find
  50. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/otal-2026-we-are-language-family-adapting/