Elamur Research⌕
☰
home / papers / 2025 / LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish

LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish

Paper · 2025 · Fred Philippy, Laura Bernardy, Siwen Guo et al.

Summary

Luxembourgish has about 400,000 speakers and very few resources for instruction tuning. Translating English instruction sets into it is a weak option, because machine translation into the language is poor and tends to produce unnatural or culturally off targets.

The authors built LuxInstruct, a cross-lingual dataset. Instructions are written in English, French or German, but every response is genuine human-written Luxembourgish taken from three sources:

  • Luxembourgish Wikipedia
  • RTL Luxembourg news
  • an online dictionary

For Wikipedia they adapted reverse instruction generation in the style of MURI. gpt-4.1-mini picks coherent spans from Luxembourgish articles and writes matching English instructions, and heuristic filters then remove bad pairs. No text is ever machine-translated into Luxembourgish.

Two experiments compared cross-lingual and monolingual instruction data. Fine-tuning with English or French instructions improved representational alignment more than Luxembourgish-only instructions. Few-shot prompts with foreign-language instructions also gave better Luxembourgish generation, as scored by three LLM judges. The work shows that a low-resource language can get native-quality targets cheaply by keeping the human text on the output side.

Key points

  • Size: LuxInstruct has 391,551 cross-lingual samples (EN/FR/DE instructions with LB outputs) plus 145,793 monolingual Luxembourgish samples.
  • Seven task types:
    • Open-Ended, from Wikipedia
    • Article-To-Title, Title-To-Article and CL-Paraphrase, from news and the LuxAlign corpus
    • Word-To-Example, Colloquial-To-Standard and Word-Translation, from the dictionary
  • Data construction: English and French news articles were paired with Luxembourgish ones using text-embedding-3-small. Wikipedia- and dictionary-derived parts are CC BY-SA 4.0, while the news part is non-commercial only.
  • Alignment experiment: alignment was measured with CKA on FLORES-200, using mean-pooled last-layer hidden states. EN-LB and FR-LB tuning gave the largest gains, and DE-LB was often below LB-LB. The authors read this as a hint that a more distant pivot language may help more than a closely related one.
  • Few-shot experiment: Gemma 3 12B/27B were tested on 50 manually verified instructions, scored with G-Eval by GPT-5 mini, Gemini 2.5 Flash-Lite and DeepSeek-V3, averaged over five seeds. English-instruction examples scored best. For Gemma 27B at 8 shots, Clarity was 81.3 with English examples versus 78.9 with Luxembourgish ones.

Interesting ideas

  • I1. Teacher understands, humans write: The LLM teacher could read Luxembourgish but not write it reliably, so it was only asked to choose spans and write the instruction, while the target stayed human-written. This keeps teacher errors off the output side.
  • I2. Pairing unaligned bilingual content by embeddings: Sentence embeddings were used to match news articles about the same story across languages, even though the outlet provides no explicit links. Comparable corpora can be mined this way where no parallel data exists.
  • I3. Representation-level diagnostic with CKA: Instead of judging only by output quality, the authors measured how closely hidden states for parallel sentences line up across languages. This shows whether training actually merged the representation spaces.
  • I4. Judge panel plus human check: Three different LLM judges were averaged, and a human annotation study confirmed that the aggregate tracks people. Repeated seeds smooth out few-shot variance.
  • I5. Translation glosses to pin down word senses: Dictionary translations were used to specify which meaning of a polysemous word an example sentence should show. This turns a lexicon into sense-specific training tasks.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Have the LLM teacher tier only write questions or labels over real human sentences, never invent the gold text. This keeps silver data anchored to authentic language.
  • I2 (back): Mined comparable pairs could act as test material for translation through the meta-world, checking whether both sides map to the same world graph.
  • I3 (tree): CKA on gparse hidden states for paraphrase pairs could reveal whether the parser encodes meaning-equivalent sentences similarly, as a check alongside LAS.
  • I4 (answer): Grade open answers on fables and tales with several judges plus a small human-verified sample, so no single judge's bias decides the score.
  • I5 (world): Cross-lingual glosses could help pick or validate the WordNet sense attached to a lemma when building the ontology layer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LuxembourgLuxembourg · Luxembourg27 works · 337 citations🎓Interdisciplinary Centre for Security, Reliability and TrustBelval · Luxembourg3 works · 3 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Fred Philippy, Laura Bernardy, Siwen Guo, Jacques Klein and Tegawende F. Bissyande. 2025. LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish. arXiv:2510.07074.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.07074 · arXiv 2510.07074

@misc{philippy2025luxinstruct,
  title = {{LuxInstruct: A Cross-Lingual Instruction Tuning Dataset For Luxembourgish}},
  author = {Fred Philippy, Laura Bernardy, Siwen Guo, Jacques Klein and Tegawende F. Bissyande},
  year = {2025},
  doi = {10.48550/arxiv.2510.07074},
  eprint = {2510.07074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.07074}
}

References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Mitiku Yohannes Fuge, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, and Seid Muhie Yimam. 2024. Walia-LLM: Enhancing Amharic-LLaMA by integrating task-specific and generative datasets. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 432–444, Miami, Florida, USA. Association for Computational Linguistics. ↗
  2. Oskar Holmström and Ehsan Doostmohammadi. 2023. Making instruction finetuning accessible to non-English languages: A case study on Swedish models. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 634– 642, Tórshavn, Faroe Islands. University of Tartu Library. find
  3. Laura Bernardy. 2022. A luxembourgish gpt-2 approach based on transfer learning. Master’s thesis, University of Trier, Trier, Germany. find
  4. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. ↗
  5. Yuri Bizzoni, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Te-Jeffrey Ip and Kritin Vongthongsri. 2025. deepeval. The Open-Source LLM Evaluation Framework. find
  6. Cedric Lothritz, Jordi Cabot, and Laura Bernardy. 2026. Testing low-resource language support in LLMs using language proficiency exams: the case of Luxembourgish. In Findings of the Association for Computational Linguistics: EACL 2026, pages 2453–2476, Rabat, Morocco. Association for Computational Linguistics.
  7. Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schuetze. 2024. LongForm: Effective instruction tuning with reverse instructions. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7056–7078, Miami, Florida, USA. Association for Computational Linguistics. find
  8. Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, and Anne Goujon. 2022. LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5080–5089, Marseille, France. European Language Resources Association. find
  9. Abdullatif Köksal, Marion Thaler, Ayyoob Imani, Ahmet Üstün, Anna Korhonen, and Hinrich Schütze. 2025. MURI: High-quality instruction tuning datasets for low-resource languages via reverse instructions. Transactions of the Association for Computational Linguistics, 13:1032–1055. ↗
  10. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3519–3529. PMLR. ↗
  11. Microsoft, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, and 56 others. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. Preprint, arXiv:2503.01743. ↗
  12. Nurkhan Laiyk, Daniil Orel, Rituraj Joshi, Maiya Goloburda, Yuxia Wang, Preslav Nakov, and Fajri Koto. 2025. Instruction tuning on public government and cultural data for low-resource language: a case study in Kazakh. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14509– 14538, Vienna, Austria. Association for Computational Linguistics. ↗
  13. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics. ↗
  14. Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, and Chengqing Zong. 2024. X-instruction: Aligning language model in low-resource languages with self-curated cross-lingual instructions. In Findings of the Association for Computational Linguistics: ACL 2024, pages 546–566, Bangkok, Thailand. Association for Computational Linguistics. find
  15. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 others. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672. ↗
  16. Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation. Preprint, arXiv:2305.15011. ↗
  17. Geyu Lin, Bin Wang, Zhengyuan Liu, and Nancy F. Chen. 2025. CrossIn: An efficient instruction tuning approach for cross-lingual knowledge alignment. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, pages 12–23, Abu Dhabi. Association for Computational Linguistics. ↗
  18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc. ↗
  19. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.
  20. Fred Philippy, Siwen Guo, Jacques Klein, and Tegawende Bissyande. 2025. LuxEmbedder: A cross-lingual approach to enhanced Luxembourgish sentence embeddings. In Proceedings of the 31st International Conference on Computational Linguistics, pages 11369–11379, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  21. Sicheng Yu, Qianru Sun, Hao Zhang, and Jing Jiang. 2022. Translate-train embracing translationese artifacts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 362–370, Dublin, Ireland. Association for Computational Linguistics. find
  22. Alistair Plum, Tharindu Ranasinghe, and Christoph Purschke. 2025. Text generation models for Luxembourgish with limited data: A balanced multilingual strategy. In Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 93–104, Abu Dhabi, UAE. Association for Computational Linguistics. ↗
  23. Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. 2024. Multilingual instruction tuning with just a pinch of multilinguality. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2304–2317, Bangkok, Thailand. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/philippy-2025-luxinstruct-cross-lingual-instruction-tuning/