Elamur Research⌕
☰
home / papers / 2023 / A Lightweight Method to Generate Unanswerable Questions in English

A Lightweight Method to Generate Unanswerable Questions in English

Paper · 2023 · Vagrant Gautam, Miaoran Zhang and Dietrich Klakow

Summary

Extractive QA systems tend to produce a plausible-looking span even when the passage does not contain the answer. Training on unanswerable examples helps, but writing them by hand is costly, and earlier automatic generators such as UNANSQ and CRQDA rely on trained neural models. The authors argue that this complexity is unnecessary. They generate unanswerable questions by making one small edit to an answerable seed question. The edit either replaces a noun, verb or adjective with a WordNet antonym, or replaces a named entity with another entity of the same type taken from the same context. Simple syntactic filters remove cases where the edit would likely leave the question answerable.

The method needs no training. They test it with six encoder models on SQuAD 2.0 and the English part of TydiQA-MinSpan. Its augmented data beats both earlier generators on downstream F1, and human judges rate the questions as more related and more readable. The study shows that cheap heuristics can serve as a strong baseline, and that larger models should be compared against such baselines before being adopted.

Key points

  • Comparison with CRQDA: on SQuAD 2.0 dev with BERT-large, entity swaps reach 83.6 F1 / 80.7 EM. CRQDA gets 82.0 F1, UNANSQ 81.0 and no augmentation 81.2. That is +1.6 F1 over the previous best.

  • TydiQA-MinSpan (English): with BERT-large, augmentation gives +9.3 F1 over no augmentation.

  • Human evaluation (3 annotators):

    SourceUnanswerabilityRelatednessReadability (1–3)
    Swaps (this paper)0.780.972.69
    CRQDA0.910.611.40
    UNANSQ0.560.982.46
    Human crowdworkers0.830.952.91

    The authors estimate roughly 20% of their generated questions are noisy, i.e. still answerable.

  • Tooling: spaCy for POS tagging and dependency parsing, an AllenNLP NER model, and NLTK's WordNet for antonyms and lemmas. GPT-2 perplexity picks the best antonym variant. The whole method is under 150 lines of Python, while CRQDA has 593M parameters.

  • Data size: for SQuAD 2.0 the method produced 34,180 antonym and 47,624 entity questions. For TydiQA it produced 880 and 2,808. Code and data are on GitHub (uds-lsv).

Interesting ideas

  • I1. A single, targeted swap can turn a question the text supports into a closely related one it cannot answer, which gives cheap contrastive negatives that stay fluent and on-topic.
  • I2. Edits are guarded with syntax: questions starting with an AUX tag (polar or alternative) are skipped, and adjectives attached to the question word are left alone, so the edit does not keep the question answerable.
  • I3. When several edited variants exist, a language model's perplexity picks the most natural one, for example "unpopular" over an awkward verb antonym.
  • I4. Taking the replacement entity from the same document and of the same type keeps the question relevant and the expected answer type unchanged. The paper's failure cases, such as commutative marriage, conjunctions and facts mentioned elsewhere, show where surface heuristics break.
  • I5. The authors found that off-the-shelf POS tagging and parsing performed noticeably worse on questions than on statements, because treebanks contain few questions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Swapping in antonyms or same-type entities from the story could produce "no answer" variants of fable and tale QA items, checking that readers abstain instead of guessing.
  • I2 (learn): The same UD-based guards could gate automatically built negative controls, keeping only perturbations whose structure does not trivially preserve the answer.
  • I3 (back): After generating several surface forms from the meaning graph, a fluency score could rank them while the graph still guarantees the content.
  • I4 (world): An explicit world of entities and symmetric relations could verify whether a swapped question really has no support, cutting the roughly 20% noise that heuristics leave.
  • I5 (tree): The perceptron and gparse parsers could be evaluated separately on interrogatives, with question-rich silver data added if accuracy lags.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany149 works · 18,349 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Vagrant Gautam, Miaoran Zhang and Dietrich Klakow. 2023. A Lightweight Method to Generate Unanswerable Questions in English. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.491

@inproceedings{gautam2023lightweight,
  title = {{A Lightweight Method to Generate Unanswerable Questions in English}},
  author = {Vagrant Gautam, Miaoran Zhang and Dietrich Klakow},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.491},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.491}
}

References

35 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6168–6173, Florence, Italy. Association for Computational Linguistics. ↗
  2. Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the AI: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662–678. ↗
  3. Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021. Improving question answering model robustness with synthetic adversarial data generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8830–8848, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  4. Rajesh Bhatt and Veneeta Dayal. 2020. Polar question particles: Hindi-urdu kya:. Natural Language & Linguistic Theory, 38(4):1115–1144. ↗
  5. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python, 1st edition. O’Reilly Media, Inc. find
  6. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 845–855, Melbourne, Australia. Association for Computational Linguistics. ↗
  7. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. The MIT Press. ↗
  10. Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1–13, Hong Kong, China. Association for Computational Linguistics.
  11. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6, Melbourne, Australia. Association for Computational Linguistics.
  12. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  13. Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, and Dongsheng Li. 2019. Read + verify: Machine reading comprehension with unanswerable questions. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6529–6537. find
  14. Yunjie Ji, Liangyu Chen, Chenxiao Dou, Baochang Ma, and Xiangang Li. 2022. To answer or not to answer? improving machine reading comprehension model with span-based contrastive learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1292–1300, Seattle, United States. Association for Computational Linguistics. find
  15. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  16. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics. ↗
  17. Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684– 5696, Online. Association for Computational Linguistics. find
  18. Najoung Kim, Phu Mon Htut, Samuel R. Bowman, and 2 Jackson Petty. 2023. (QA) : Question answering with questionable assumptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8466–8487, Toronto, Canada. Association for Computational Linguistics. find
  19. Jonathan K. Kummerfeld. 2021. Quantifying and avoiding unfair qualification labour in crowdsourcing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 343–349, Online. Association for Computational Linguistics. ↗
  20. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. ↗
  21. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
  22. Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Jiancheng Lv, Nan Duan, and Ming Zhou. 2020. Tell me how to ask again: Question data augmentation with controllable rewriting in continuous space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5798–5810, Online. Association for Computational Linguistics. find
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  24. Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756–1765, Vancouver, Canada. Association for Computational Linguistics.
  25. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  26. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics. ↗
  27. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  28. Saku Sugawara, Nikita Nangia, Alex Warstadt, and Samuel Bowman. 2022. What makes reading comprehension questions difficult? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6951–6971, Dublin, Ireland. Association for Computational Linguistics. find
  29. Elior Sulem, Jamaal Hay, and Dan Roth. 2021. Do we know what we don’t know? studying unanswerable questions beyond SQuAD 2.0. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4543–4548, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  30. Chuanqi Tan, Furu Wei, Qingyu Zhou, Nan Yang, Weifeng Lv, and Ming Zhou. 2018. I know there is no answer: Modeling answer validation for machine reading comprehension. In Natural Language Processing and Chinese Computing, pages 85–97, Cham. Springer International Publishing. find
  31. Yicheng Wang and Mohit Bansal. 2018. Robust machine comprehension models via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 575–581, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  32. Hyperparameter Learning Rate Batch Size Epochs Max Seq Length Doc Stride Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 271–280, Vancouver, Canada. Association for Computational Linguistics. ↗
  33. Vanessa Williamson. 2016. On the ethics of crowdsourced research. PS: Political Science & Politics, 49(1):77–81. ↗
  34. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  35. Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651–1661, Florence, Italy. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gautam-2023-lightweight-method-generate-unanswerable-questions/