Elamur Research⌕
☰
home / papers / 2026 / Semantic, Syntactic, Lexical: What Makes QA Augmentation Work in Limited Quantity?

Semantic, Syntactic, Lexical: What Makes QA Augmentation Work in Limited Quantity?

Paper · 2026 · Benedictus Kent Rachmat, Thomas Gerald, Takuya Nakamura et al.

Summary

The paper asks which kind of synthetic variation actually helps question answering models when labelled data is scarce. It also asks whether such data stays faithful to the passage and keeps the original answerability. The authors build small nested training subsets of SQuADv2 (general domain) and PubMedQA (biomedical) to simulate low-resource conditions.

From each question, Llama-3.1-70B-Instruct produces three families of variants:

  • lexical: synonym swaps;
  • syntactic: structural rewrites;
  • semantic: new questions grounded in the same passage.

GPT-4o screens these variants, and a human audit cross-checks the screening. Llama8B-I and Qwen7B-I are then fine-tuned with LoRA and scored with EM/F1 plus quality diagnostics.

On SQuADv2, augmented sets roughly match the baselines, so the extra diversity rarely hurts but seldom helps. For PubMedQA, only semantic augmentation delivers reliable improvements at the smallest data scale. Gains do not grow steadily with data size, which the authors link to LoRA dynamics. The study argues that judging augmentation requires quality checks, not accuracy alone.

Key points

  • Generation: each original question receives two variants per augmentation type, yielding six synthetic counterparts in total.

  • Validation outcomes: GPT-4o kept these counts after filtering:

    DatasetValidUnsureInvalid
    SQuADv217,76847,207
    PubMedQA19,050175,927

  • Supervision scales: augmented training uses 1/128, 1/64 and 1/32 of the original data, with 1/16 used for the baseline only. SQuADv2 subsets range from 1,015 to 8,125 samples. Fixed test sets contain 11,873 items (SQuADv2) and 1,000 items (PubMedQA).

  • Judge vs. human audit: the judge's leniency depends on the domain.

    • On SQuADv2, human annotators accepted 81.5% of items versus 77.3% for GPT-4o, so the two broadly agree.
    • On PubMedQA, humans accepted 66.4% versus 79.2% for GPT-4o, showing the model is notably more permissive on biomedical text.

  • Compute and energy: energy was tracked with CodeCarbon on 2× NVIDIA H100 GPUs, and the choice of augmentation type barely affected cost. Qwen7B-I was faster on SQuADv2 but slower on PubMedQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓Université Paris-SaclayGif-sur-Yvette · France45 works · 6,127 citations🔬Laboratoire Interdisciplinaire des Sciences du NumériqueGif-sur-Yvette · France14 works · 507 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 441 works

Cite

Cite as Benedictus Kent Rachmat, Thomas Gerald, Takuya Nakamura, Zheng Zhang and Cyril Grouin. 2026. Semantic, Syntactic, Lexical: What Makes QA Augmentation Work in Limited Quantity?. Proceedings of the Workshop on Structured Linguistic Data and Evaluation (SLiDE).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2545w4h6dty8

@inproceedings{rachmat2026semantic,
  title = {{Semantic, Syntactic, Lexical: What Makes QA Augmentation Work in Limited Quantity?}},
  author = {Benedictus Kent Rachmat, Thomas Gerald, Takuya Nakamura, Zheng Zhang and Cyril Grouin},
  booktitle = {Proceedings of the Workshop on Structured Linguistic Data and Evaluation (SLiDE)},
  year = {2026},
  doi = {10.63317/2545w4h6dty8},
  url = {https://doi.org/10.63317/2545w4h6dty8}
}

References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6168–6173, Florence, Italy. Association for Computational…
  2. Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott Mc-Carley, Michael McCawley, Mohamed Nasr, Lin Pan, Cezar Pendus, John Pitrelli, Saurabh Pujar, Salim Roukos, Andrzej Sakrajda, Avi Sil, Rosario Uceda-Sosa, Todd W… find
  3. Yung-Chieh Chan, George Pu, Apaar Shanker, Parth Suresh, Penn Jenks, John Heyer, and Sam Denton. 2024. Balancing cost and effectiveness of synthetic data generation strategies for llms. arXiv preprint arXiv:2409.19759. — [arXiv:2409.19759](https://arxiv.org/abs/2409.19759) find
  4. Jie Chen, Yupeng Zhang, Bingning Wang, Wayne Xin Zhao, Ji-Rong Wen, and Weipeng Chen. 2024. Unveiling the flaws: Exploring imperfections in synthetic data and mitigation strategies for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14855–14865,… find
  5. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural…
  6. Arijit Chowdhury and Aman Chadha. 2024. Generative data augmentation using llms improves distributional robustness in question answering. In Proceedings of the 18th conference of the European chapter of the association for computational linguistics: student research workshop, pages 258–265. find
  7. Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, MarionCoutarel, Boris Feld, Jérémy Lecourt, LiamConnell, Amine Saboni, Inimaz, supatomic, Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Hugues de Lavoreille, Niko Laskaris, Edoardo… find
  8. Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Fang Zeng, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2025. Auggpt: Leveraging chatgpt for text data augmentation.…
  9. Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988, Online. Association for Computational Lin…
  10. Dan Friedman and Adji Bousso Dieng. 2023. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research. find
  11. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on LLM-as-a-Judge. find
  12. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessic… — LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  13. Zhen Guo, Peiqi Wang, Yanwei Wang, and Shangdi Yu. 2023. Improving small language models on pubmedqa via generative data augmentation. In LLM4AI’23: Workshop on Foundations and Applications in Large-scale AI Models -Pre-training, Fine-tuning, and Prompt-based Learning, Long Beach, CA, USA. find
  14. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. ArXiv preprint arXiv:2203.15556.
  15. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. pages 2567–2577, Hong Kong, China. Association for Computational Linguistics.
  16. Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. 2022. Biomedical question answering: A survey of approaches and challenges. ACM Comput. Surv., 55(2). find
  17. Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova Dassarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Her… find
  18. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. 2025. Why language models hallucinate. ArXiv preprint 2509.04664v1. — [arXiv:2509.04664](https://arxiv.org/abs/2509.04664) find
  19. Julien Khlaut, Corentin Dancette, Elodie Ferreres, Alaedine Bennani, Paul Hérent, and Pierre Manceron. 2024. Efficient medical question answering with knowledge-augmented question generation. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 10–20, Mexico City, Mexico. A… find
  20. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
  21. Qin Liu, Fei Wang, Nan Xu, Tianyi Lorena Yan, Tao Meng, and Muhao Chen. 2024. Monotonic paraphrasing improves generalization of language model prompting. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9861–9877, Miami, Florida, USA. Association for Computational Ling… find
  22. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. Technical report, OpenAI.
  23. Shayne Longpre, Yi Lu, Zhucheng Tu, and Chris DuBois. 2019. An exploration of data augmentation and sampling techniques for domain-agnostic question answering. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 220–227, Hong Kong, China. Association for Computational… find
  24. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  25. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. Green ai. Commun. ACM, 63(12):54–63.
  26. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2021. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003. find
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
  28. Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. 2025. Concise thoughts: Impact of output length on LLM reasoning and cost. ArXiv preprint arXiv:2407.19825. — [arXiv:2407.19825](https://arxiv.org/abs/2407.19825) find
  29. Sergey Pletenev, Maria Marina, Daniil Moskovskiy, Vasily Konovalov, Pavel Braslavski, Alexander Panchenko, and Mikhail Salnikov. 2025. How much knowledge can you pack into a lora adapter without harming llm? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4309–4322. find
  30. Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020. Training question answering models from synthetic data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5811–5826, Online. Association for Computational Ling…
  31. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling…
  32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. Association for Computational Linguistics.
  33. Revanth Gangi Reddy, Bhavani Iyer, Md Arafat Sultan, Rong Zhang, Avi Sil, Vittorio Castelli, Radu Florian, and Salim Roukos. 2020. End-to-end QA on COVID-19: Domain adaptation with synthetic training. ArXiv preprint arXiv:2012.01414. — [arXiv:2012.01414](https://arxiv.org/abs/2012.01414) find
  34. Maximilian Schmidt, Andrea Bartezzaghi, and Ngoc Thang Vu. 2024. Prompting-based synthetic data generation for few-shot question answering. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13168–13178. find
  35. Minju Seo, Jinheon Baek, James Thorne, and Sung Ju Hwang. 2024. Retrieval-augmented data augmentation for low-resource domain tasks. ArXiv preprint arXiv:2402.13482v1. — [arXiv:2402.13482](https://arxiv.org/abs/2402.13482) find
  36. Siamak Shakeri, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. End-to-end synthetic data generation for domain adaptation of question answering systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language… find
  37. Marco Antonio Sobrevilla Cabezudo, Marcio Lima Inacio, and Thiago Alexandre Salgueiro Pardo. 2024. Investigating paraphrase generation as a data augmentation strategy for low-resource AMR-to-text generation. In Proceedings of the 17th International Natural Language Generation Conference, pages 663–6… find
  38. Emma Strubell, Ananya Ganesh, and Andrew Mc-Callum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  39. Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326–346, Online. Association for Computational Linguistics.
  40. Kanix Wang, Robert Stevens, Halima Alachram, Yu Li, Larisa Soldatova, Ross King, Sophia Ananiadou, Maolin Li, Fenia Christopoulou, Jose Luis Ambite, Joel Matthew, Sahil Garg, Ulf Hermjakob, Daniel Marcu, Emily Sheng, Tim Beißbarth, Edgar Wingender, Aram Galstyan, and Andrey Rzhetsky. 2021. Nero: a b… find
  41. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  42. Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP…
  43. Joonho Yang, Seunghyun Yoon, Hwan Chang, Byeongjeong Kim, and Hwanhee Lee. 2025. Hallucinate at the last in long response generation: A case study on long document summarization. ArXiv preprint arXiv:2505.15291v2. — [arXiv:2505.15291](https://arxiv.org/abs/2505.15291) find
  44. Yuchang Zhu, Huizhe Zhang, Bingzhe Wu, Jintang Li, Zibin Zheng, Peilin Zhao, Liang Chen, and Yatao Bian. 2025. Measuring diversity in synthetic datasets. In Forty-second International Conference on Machine Learning. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rachmat-2026-semantic-syntactic-lexical-what-makes/