Elamur Research⌕
☰
home / papers / 2023 / ELQA: A Corpus of Metalinguistic Questions and Answers about English

ELQA: A Corpus of Metalinguistic Questions and Answers about English

Paper · 2023 · Shabnam Behzad, Keisuke Sakaguchi, Nathan Schneider et al.

Summary

Metalanguage is language that takes language itself as its topic, for example claims about what a phrase means or which form is grammatical. NLP datasets rarely target it. The authors built ELQA, a corpus of more than 70k questions with their answers, drawn from two Stack Exchange communities: ENG (English Language & Usage) and ELL (English Language Learners). It is released in two versions. ELQA-large uses light filtering. ELQA-small uses stricter quality filtering and was checked by human annotators. The authors also sorted the questions into a manual taxonomy of question types. As a first case study, they framed a closed-book, free-form answer generation task and tested T5 and GPT-3 configurations, using both automatic metrics and human judges. The models' answers were mostly fluent and well formed. Their validity, however, was clearly below that of human-written answers, which shows that explaining language is still hard for LLMs. The resource is useful for probing metalinguistic ability and for educational NLP aimed at language learners.

Key points

  • Source and license: posts up to 2021-12-06, taken from Internet Archive dumps and released under CC-BY-SA 3.0. Each entry has title, body (HTML and plain text), tags, score, favorite count, user bio and answers.
  • ELQA-large: ENG has 47,532 questions and 152,315 answers; ELL has 23,520 questions and 49,345 answers. Questions with images were dropped, and every question needs an answer scored at least 2.
  • ELQA-small: ENG has 14,234 questions; ELL has 6,477. Filters require:
    • a question score of at least 2,
    • an answer scored above 3,
    • at least one highly rated answer with no hyperlinks.
  • Quality check: a human audit of 250 pairs judged 99.2% of questions answerable and 91.8% of answers acceptable.
  • Question taxonomy: based on 100 labelled questions, with 92% inter-annotator agreement on the overlap:
    • Fluency ≈38%
    • Meaning to Form ≈20%
    • Form to Meaning ≈19%
    • Grammatical Analysis ≈11%
    • Other (etymology, pronunciation, etc.) ≈10%

Interesting ideas

  • I1. The authors dropped reference answers that depend on external links, so a closed-book model is never judged against resources it could not have retrieved. This makes the comparison with human answers fair.
  • I2. Questions about language are split by direction: from a form to its meaning, from a meaning to a form, fluency or acceptability judgments, and structural analysis. Each direction calls for a different kind of reasoning.
  • I3. A word that is mentioned is an object with properties, not a use carrying its normal denotation. One model failure came from explaining the wrong occurrence of a repeated word ("in" appeared twice).
  • I4. Human evaluation scored well-formedness and validity separately, which revealed that fluent output can still be wrong.
  • I5. The authors caution that a correct answer may simply echo frequently repeated web topics rather than show abstraction. They also argue that confidence estimation is needed before showing answers to users.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when building gold answers for evaluation, filter out items whose correct answer needs information outside the text and Mova's compiled knowledge, so failures reflect reasoning rather than missing access.
  • I2 (answer): a classifier for metalinguistic questions could send each one to a different reader, e.g. a lexicon lookup for meaning, the generator for encoding, and the parser itself for grammatical-analysis questions.
  • I3 (world): mentioned expressions such as quoted or bolded strings could become linguistic entities tied to an exact token span. This keeps references unambiguous and stops them leaking into the event semantics.
  • I4 (back): rate generated text on two separate axes, surface fluency and faithfulness to the meaning graph, so that a smooth but wrong compression is not counted as success.
  • I5 (learn): add negative controls that compare common forum-style questions with unusual constructed ones to detect memorisation. Calibrated confidence could then decide which answers Mova presents without caveats.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States88 works · 4,732 citations🎓Tohoku UniversitySendai · Japan20 works · 514 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Shabnam Behzad, Keisuke Sakaguchi, Nathan Schneider and Amir Zeldes. 2023. ELQA: A Corpus of Metalinguistic Questions and Answers about English. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.113

@inproceedings{behzad2023elqa,
  title = {{ELQA: A Corpus of Metalinguistic Questions and Answers about English}},
  author = {Shabnam Behzad, Keisuke Sakaguchi, Nathan Schneider and Amir Zeldes},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.113},
  url = {https://doi.org/10.18653/v1/2023.acl-long.113}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arshad Ahmad, Chong Feng, Shi Ge, and Abdallah Yousif. 2018. A survey on mining stack overflow: question and answering (Q&A) community. Data Technol. Appl., 52:190–247. ↗
  2. Michael L. Anderson, Andrew Fister, Bryant Lee, Luwito Tardia, and Danny Wang. 2004. On the types and frequency of meta-language in conversation: A preliminary report. In 14th Annual Meeting of the Society for Text and Discourse. find
  3. Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. 2021. Think you have solved direct-answer question answering? Try ARC-DA, the direct-answer AI2 reasoning challenge. arXiv preprint arXiv:2102.03315. ↗
  4. Ksenija Bogetic. 2021. MetaLangCORP: Presenting the first corpus of media metalanguage in Slovene, Croatian and Serbian, and its cross-discipline applicability. Fluminensia, 33:123–142. ↗
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy. Association for Computational Linguistics.
  7. Jill Burstein. 2003. The E-rater® scoring engine: Automated essay scoring with natural language processing. In Automated essay scoring: A cross-disciplinary perspective, pages 113–121. Lawrence Erlbaum Associates Publishers. find
  8. Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Deriu, Mark Cieliebak, and Eneko Agirre. 2020. DoQA - accessing domain-specific FAQs via conversational QA. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7302–7314, Online. Association for Computational Linguistics. find
  9. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. ArXiv, abs/2006.14799. ↗
  10. Maria Chinkina and Detmar Meurers. 2017. Question generation for language learning: From ensuring texts are read to supporting learning. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 334–344, Copenhagen, Denmark. Association for Computational Linguistics. find
  11. Robert Dale, Ilya Anisimoff, and George Narroway. 2012. HOO 2012: A report on the preposition and determiner error correction shared task. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 54–62, Montréal, Canada. Association for Computational Linguistics.
  12. Cícero dos Santos, Luciano Barbosa, Dasha Bogdanova, and Bianca Zadrozny. 2015. Learning hybrid representations to retrieve semantically equivalent questions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 694–699, Beijing, China. Association for Computational Linguistics. find
  13. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics. ↗
  14. Youmna Farag, Helen Yannakoudakis, and Ted Briscoe. 2018. Neural automated essay scoring and coherence modeling for adversarially crafted input. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 263–271, New Orleans, Louisiana. Association for Computational Linguistics. find
  15. Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015. CQADupStack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian Document Computing Symposium (ADCS), ADCS ’15, pages 3:1– 3:8, New York, NY, USA. ACM. find
  16. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 9:962–977. ↗
  17. Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, and Noah A. Smith. 2022. Bidimensional leaderboards: Generate and evaluate language hand in hand. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3540–3557, Seattle, United States. Association for Computational Linguistics. ↗
  18. Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021. GooAQ: Open question answering with diverse answer types. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 421–433, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  19. Vaibhav Kumar and Alan W Black. 2020. ClarQ: A large-scale and diverse dataset for clarification question generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7296–7301, Online. Association for Computational Linguistics. find
  20. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. ↗
  21. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  22. Edison Marrese-Taylor, Ai Nakajima, Yutaka Matsuo, and Ono Yuichi. 2018. Learning to automatically generate fill-in-the-blank quizzes. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 152–156, Melbourne, Australia. Association for Computational Linguistics. find
  23. Preslav Nakov, Doris Hoogeveen, Lluís Màrquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. SemEval-2017 Task 3: Community question answering. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 27–48, Vancouver, Canada. Association for Computational Linguistics.
  24. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14, Baltimore, Maryland. Association for Computational Linguistics. find
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  26. Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019. Introducing MANtIS: a novel multi-domain information seeking dialogues dataset. arXiv preprint arXiv:1912.04639. ↗
  27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  29. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393–401. ↗
  30. Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. 2022. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203.17189. ↗
  31. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. ↗
  32. Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. QA dataset explosion: A taxonomy of NLP resources for question answering and reading comprehension. ACM Comput. Surv., 55(10). ↗
  33. Yuan Yao, Hanghang Tong, Tao Xie, Leman Akoglu, Feng Xu, and Jian Lu. 2013. Want a good answer? Ask a good question first! arXiv preprint arXiv:1311.6876. ↗
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. ↗
  35. Keisuke Sakaguchi, Yuki Arase, and Mamoru Komachi. 2013. Discriminative approach to fill-in-the-blank quiz generation for language learners. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238–242, Sofia, Bulgaria. Association for Computational Linguistics. find
  36. Charlotte Taylor. 2015. Beyond sarcasm: The metalanguage and structures of mock politeness. Journal of Pragmatics, 87:127–141. ↗
  37. Kristiyan Vachev, Momchil Hardalov, Georgi Karadzhov, Georgi Georgiev, Ivan Koychev, and Preslav Nakov. 2021. Generating answer candidates for quizzes and answer-aware question generators. In Proceedings of the Student Research Workshop Associated with RANLP 2021, pages 203–209, Online. INCOMA Ltd. find
  38. Yu Wang, Yuelin Wang, Kai Dang, Jie Liu, and Zhuo Liu. 2021. A comprehensive survey of grammatical error correction. ACM Trans. Intell. Syst. Technol., 12(5). ↗
  39. Shomir Wilson. 2010. Distinguishing use and mention in natural language. In Proceedings of the NAACL HLT 2010 Student Research Workshop, pages 29–33, Los Angeles, CA. Association for Computational Linguistics. find
  40. Shomir Wilson. 2011. In search of the use-mention distinction and its impact on language processing tasks. IJCLA, 2(1-2):139–154. find
  41. Shomir Wilson. 2012. The creation of a corpus of English metalanguage. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 638–646, Jeju Island, Korea. Association for Computational Linguistics.
  42. Shomir Wilson. 2013. Toward automatic processing of English metalanguage. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 760–766, Nagoya, Japan. Asian Federation of Natural Language Processing.
  43. Shomir Wilson. 2017. A bridge from the use-mention distinction to natural language processing. In Paul Saka and Michael Johnson, editors, The Semantics and Pragmatics of Quotation, pages 79–96. Springer International Publishing, Cham. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/behzad-2023-elqa-corpus-metalinguistic-questions-answers/