Elamur Research⌕
☰
home / papers / 2026 / LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering

LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering

Paper · 2026 · Nina Hosseini-Kivanani, Marco Matassoni and Alessio Brutti

Summary

The paper asks whether spoken question answering (SQA) can be built for Luxembourgish, a low-resource language, without recording a large human QA speech corpus. The authors take existing text QA resources and machine-translate the questions into Luxembourgish. They then turn the questions into audio with several TTS engines and pair each spoken question with its textual answer. On this data they train a SLAM-style speech-LLM. A frozen Whisper encoder feeds frozen multilingual LLMs (EuroLLM, Apertus) through a trained linear projector plus LoRA adapters, and the model is supervised directly on question→answer pairs, not on transcription. Testing on LLAMA-LB-Test, which has two real Luxembourgish speaker conditions, showed that mixing several TTS sources and using voice-design synthesis gave the best results. A further finding is that no-reference TTS quality scores do not reliably rank which synthetic corpus will train the better QA model. Synthetic audio should therefore be judged by its value as training data, not by how natural it sounds. The authors frame the workflow as transferable to other low-resource languages.

Key points

  • Architecture: frozen Whisper large v3 turbo encoder → trainable linear projector → frozen EuroLLM or Apertus backend with LoRA adapters. The model goes straight from speech to a text answer, with no ASR cascade.
  • TTS engines compared: MMS-TTS, Qwen3-TTS (CustomVoice cloning and VoiceDesign from text descriptions) and OmniVoice (cloning and voice design).
  • Corpus sizes: single-source corpora have roughly 48k questions each, and the combined "4TTS" multi-source mix has about 230k questions.
  • Datasets listed for training and evaluation include SQuAD-LB-TTS, LuxASR-50K, LuxASR-TTS-50K, HeySQuAD and LLAMA-EN. Final testing uses LLAMA-LB-Test with real speakers.
  • No-reference TTS quality metrics are not monotonic predictors of downstream SQA accuracy.

Interesting ideas

  • I1. Measure synthetic data by what it does downstream rather than by surface quality: a corpus that "sounds" better did not always train a better model.
  • I2. Mixing outputs from several generators (four TTS systems) beats relying on any single one, because the model cannot overfit to one generator's quirks.
  • I3. Train entirely on synthetic material but evaluate only on genuine human input under distinct conditions, which exposes the synthetic-to-real gap honestly.
  • I4. Voice design creates new, varied voices from a textual description, giving diversity without needing any reference samples.
  • I5. Supervising the connector directly on the end task (question→answer) instead of an intermediate task (transcription) aligns the learned component with what actually matters.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver trees agreed by witness parsers could be ranked by how much they improve frame and answer accuracy on held-out tests, not only by parser agreement or confidence.
  • I2 (learn): Training text produced by LLM teacher tiers could deliberately blend several teachers or prompting styles so the parser does not learn one model's habitual phrasing.
  • I3 (answer): Keep school-test and fable QA evaluation sets strictly human-written and split by source or style, while generated questions are used only for training and diagnostics.
  • I4 (back): The generator could render one meaning graph under several described style profiles (formal, childlike, terse) to produce diverse paraphrased questions for stress-testing readers.
  • I5 (frames): Learned pieces of the frame stage, such as role assignment, could be tuned against answer correctness on QA items, not only against intermediate role-label agreement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Fondazione Bruno KesslerTrento · Italy35 works · 43,945 citations🎓University of LuxembourgLuxembourg · Luxembourg28 works · 337 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Nina Hosseini-Kivanani, Marco Matassoni and Alessio Brutti. 2026. LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering. arXiv:2607.02763.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.02763 · arXiv 2607.02763

@misc{hosseinikivanani2026luxsqa,
  title = {{LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering}},
  author = {Nina Hosseini-Kivanani, Marco Matassoni and Alessio Brutti},
  year = {2026},
  doi = {10.48550/arxiv.2607.02763},
  eprint = {2607.02763},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.02763}
}

References

35 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. You, N. Chen, F. Liu, S. Ge, X. Wu, and Y. Zou, “End-to-end Spoken Conversational Question Answering: Task, Dataset and Model,” in Findings of the Association for Computational Linguistics: NAACL 2022, M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, Eds. Seattle, United States: Association for Computational Linguistics, Jul. 2022, pp. 1219–1232. [Online]. Available: https://aclanthology.org/2022.findings-naacl.91/ ↗
  2. G. Sidiropoulos, S. Vakulenko, and E. Kanoulas, “On the Impact of Speech Recognition Errors in Passage Retrieval for Spoken Question Answering,” in Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM ’22). ACM, 2022, pp. 17–21. find
  3. M. Wang, W. Han, I. Shafran, Z. Wu, C.-C. Chiu, Y. Cao, N. Chen, Y. Zhang, H. Soltau, P. K. Rubenstein et al., “SLM: Bridge the thin gap between speech and text foundation models,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8.
  4. Z. Ma, G. Yang, Y. Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhang et al., “SLAM-ASR: An Embarrassingly Simple Approach for LLM with Strong ASR Capacity,” arXiv preprint arXiv:2402.08846, 2024. ↗
  5. ——, “Speech Recognition Meets Large Language Model: Benchmarking, Models, and Exploration,” Proc. AAAI, 2025. find
  6. Z. Ma, G. Yang, W. Chen, Z. Gao, Y. Du, X. Li, Z. Zheng, H. Zhu, J. Zhuo, Z. Song et al., “SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing,” IEEE Journal of Selected Topics in Signal Processing, 2026. ↗
  7. S. Kumar, I. Thorbecke, S. Burdisso, E. Villatoro-Tello, M. KE, K. Ha-cioğlu, P. Rangappa, P. Motlicek, A. Ganapathiraju, and A. Stolcke, “Performance evaluation of slam-asr: The good, the bad, the ugly, and the way forward,” in 2025 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). IEEE, 2025, pp. 1–5. find
  8. S. Fong, M. Matassoni, and A. Brutti, “Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages,” in Proc. Interspeech 2025, 2025, pp. 2003– 2007. find
  9. C. You, N. Chen, F. Liu, D. Yang, and Y. Zou, “Towards data distillation for end-to-end spoken conversational question answering,” arXiv preprint arXiv:2010.08923, 2020. ↗
  10. M. Ü. Menevşe, Y. Manav, E. Arisoy, and A. Özgür, “Dealing with data scarcity in spoken question answering,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 4449–4455. find
  11. ——, “A framework for automatic generation of spoken question-answering data,” in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 4659–4666. find
  12. Y. Wu, S. Rallabandi, R. Srinivasamurthy, P. P. Dakle, A. Gon, and P. Raghavan, “HeySQuAD: A Spoken Question Answering Dataset,” 2023. ↗
  13. Z. Zhao, Y. Jiang, H. Liu, Y. Wang, and Y. Wang, “LibriSQA: A novel dataset and framework for spoken question answering with large language models,” IEEE Transactions on Artificial Intelligence, 2024. ↗
  14. N. Chen, C. You, and Y. Zou, “Self-supervised dialogue learning for spoken conversational question answering,” in Proceedings of INTER-SPEECH, 2021. find
  15. W. Chen, Z. Ma, R. Yan, Y. Liang, X. Li, R. Xu, Z. Niu, Y. Zhu, Y. Yang, Z. Liu, K. Yu, Y. Hu, J. Li, Y. Lu, S. Liu, and X. Chen, “SLAM-Omni: Timbre-Controllable Voice Interaction System with Single-Stage Training,” in Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for Computational Linguistics, Jul. 2025, pp. 2262–2282. [Online]. Available: https://aclanthology.org/2025.findings-acl.115/ ↗
  16. M.-H. Shih, H.-L. Chung, Y.-C. Pai, M.-H. Hsu, G.-T. Lin, S.-W. Li, and H.-y. Lee, “GSQA: An End-to-End Model for Generative Spoken Question Answering,” in Proc. Interspeech 2024, 2024, pp. 2970–2974. find
  17. E. Nachmani, A. Levkovitch, R. Hirsch et al., “Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM,” in Proceedings of the International Conference on Learning Representations (ICLR), 2024. ↗
  18. D. Wang, J. Xu, R. Chu, Z. Guo, X. Wang, J. Wu, D. Yang, S. Ji, and J. Lin, “Inserter: Speech instruction following with unsupervised interleaved pre-training,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 18 024–18 046. find
  19. R. Zevallos, N. Bel, G. Cámbara, M. Farrús, and J. Luque, “Data Augmentation for Low-Resource Quechua ASR Improvement,” in Proc. Interspeech 2022, 2022, pp. 3518–3522. ↗
  20. E. Casanova, S. Aluı́sio, and M. A. Ponti, “TTS applied to the generation of datasets for automatic speech recognition,” in Proceedings of the 16th International Conference on Computational Processing of Portuguese-Vol. 1, 2024, pp. 633–638. find
  21. C. Mi, L. Xie, and Y. Zhang, “Improving data augmentation for low resource speech-to-text translation with diverse paraphrasing,” Neural Networks, vol. 148, pp. 194–205, 2022. ↗
  22. C.-J. Lin, G.-T. Lin, Y.-S. Chuang, W.-L. Wu, S.-W. Li, A. Mohamed, H.-y. Lee, and L.-s. Lee, “SpeechDPR: End-to-end spoken passage retrieval for open-domain spoken question answering,” arXiv preprint arXiv:2401.13463, 2024. ↗
  23. Z. Zhao, Y. Jiang, H. Liu, Y. Wang, and Y. Wang, “LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models,” IEEE Transactions on Artificial Intelligence, 2023. ↗
  24. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli, “Scaling speech technology to 1,000+ languages,” J. Mach. Learn. Res., vol. 25, no. 1, Jan. 2024. ↗
  25. H. Hu, X. Zhu, T. He, D. Guo, B. Zhang, X. Wang, Z. Guo, Z. Jiang, H. Hao, Z. Guo, X. Zhang, P. Zhang, B. Yang, J. Xu, J. Zhou, and J. Lin, “Qwen3-tts technical report,” arXiv preprint arXiv:2601.15621, 2026. ↗
  26. H. Zhu, L. Ye, W. Kang, Z. Yao, L. Guo, F. Kuang, Z. Han, W. Zhuang, L. Lin, and D. Povey, “OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models,” arXiv preprint arXiv:2604.00688, 2026. ↗
  27. P. Gilles, N. Hosseini-Kivanani, and L. E. A. Hillah, “LUX-ASR: Building an ASR system for the Luxembourgish language,” in 2022 IEEE Spoken Language Technology Workshop (SLT) SLT 2022, 2023. find
  28. Y. Wu, S. Rallabandi, R. Srinivasamurthy, P. P. Dakle, A. Gon, and P. Raghavan, “HeySQuAD: A Spoken Question Answering Dataset,” arXiv preprint arXiv:2304.13689, 2024. ↗
  29. G. Mittag and S. Möller, “Deep Learning Based Assessment of Synthetic Speech Naturalness,” Interspeech 2020, 2020. find
  30. C. K. Reddy, V. Gopal, and R. Cutler, “DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6493–6497. find
  31. P. H. Martins et al., “EuroLLM: Multilingual Language Models for Europe,” Procedia Computer Science, vol. 255, pp. 53–62, 2025, proceedings of the Second EuroHPC user day. ↗
  32. A. Hernández-Cano, A. Hägele, A. H. Huang, A. Romanou, A.-J. Solergibert, B. Pasztor, B. Messmer, D. Garbaya, E. F. Ďurech, I. Hakimi et al., “Apertus: Democratizing open and compliant llms for global language environments,” in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026, pp. 46 877–46 955.
  33. N. Rossenbach, S. Sakti, and R. Schlüter, “On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition,” in Proc. SynData4GenAI 2024, 2024, pp. 21–25. find
  34. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023.
  35. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: Nlg evaluation using gpt-4 with better human alignment,” in Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 2511–2522.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hosseini-kivanani-2026-luxsqa-ask-me-luxembourgish-tts/