Large-Scale QA-SRL Parsing
What it is about
The paper releases QA-SRL Bank 2.0, which has 265,140 valid question-answer pairs for 64,018 sentences and 133,479 verbs across Wikipedia, Wikinews and science textbooks. A streamlined crowdsourcing interface, with an autocomplete automaton over the seven question slots and a separate validation stage with two validators, was used; the whole set took nine days and cost $43,647, or 16.5 cents per valid question. The parser is a pipeline: span detection (a BIO tagger, or a span-based model that scores every span) followed by question generation (a local slot classifier, or a sequential LSTM decoder). The final parser reaches 82.6% question accuracy and 77.6% span accuracy in a human evaluation. Re-running the parser to over-generate questions and validating them with the crowd adds 20% valid questions to the training and development sets.
Ideas
- I1. The seven-slot question template (WH, AUX, SBJ, TGT, OBJ, PREP, MISC) as a deterministic grammar for school questions in
world/src/qa.rs. The paper's autocomplete automaton is a grammaticality filter, and it can be written as a finite-state check in Rust. - I2. Span detection from UD subtrees: candidate answers are the subtrees of the verb's dependents, scored by a count-based rule, and kept above a threshold. Measure span F1 on dev with the paper's IOU of at least 0.5.
- I3. Data expansion as a deterministic loop: generate candidate question-answer pairs from the parser, keep those not already covered by an answer, and validate them by a second, independent source. Keep the held-out test untouched.
- I4. Negative control: shuffle the question slots of correct questions. Question accuracy must drop, otherwise the slots are not doing any work.
For Mova
The chapters are languages/en/26-questions.md and languages/en/13-verbs.md. The code is world/src/qa.rs and en/src/parse.rs for the subtrees. The measures are span F1 at IOU 0.5 on dev, question accuracy against a human-checked sample, and the negative control. Run each twice.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1191.pdf
Summary
Semantic role labeling (SRL) resources such as PropBank and FrameNet depend on expert annotators with long guidelines, which makes large training sets expensive. This paper shows that QA-SRL can be scaled through crowdsourcing. In QA-SRL, each predicate-argument relation is represented as a templated natural-language question about a verb, paired with answer spans from the sentence. The authors built a two-stage pipeline on Amazon Mechanical Turk:
- Generation: workers write questions using an NFA-driven autocomplete and auto-suggest interface.
- Validation: separate workers answer each question or reject it.
The result is QA-SRL Bank 2.0, covering Wikipedia, Wikinews and science textbook text. They then trained the first strong QA-SRL parser, a syntax-free pipeline with two steps: span-based argument detection, then an LSTM decoder that generates a slot-structured question for each detected span. Finally, they ran a recall-oriented version of the parser and filtered its extra questions through the validation stage, which enlarged the data and improved the retrained models. Under human evaluation, the final system reaches roughly 83% question accuracy and 78% span accuracy. This makes cheap, interpretable predicate-argument annotation feasible at scale.
Key points
- Size: QA-SRL Bank 2.0 has 265,140 valid question-answer pairs for 133,479 verbs in 64,018 sentences (1.27M tokens). The domains are Wikipedia, Wikinews and TQA science textbooks.
- Cost and speed: Collection took 9 days with 1,165 workers and cost about $43.6K. That is roughly 32.7¢ per verb, compared with about 50¢ per verb for the original hired-annotator QA-SRL data (He et al., 2015).
- Annotation quality: On a pooled 100-verb subset, the crowdsourced data scored P 95.7 / R 72.4 / F 82.4. The original dataset scored P 97.5 / R 86.6. Validity judgments showed moderate agreement (Fleiss's Kappa about 0.51–0.55).
- Parser scores against crowd labels: Up to 82.2% span-detection F1 and 47.2% exact-match question accuracy. Under human evaluation at about 2 questions per verb, question accuracy is 82.6% and span accuracy is 77.6%.
- Data expansion: Over-generating with the parser and then validating grew the train and dev sets by about 20%. Data, code and models are released at qasrl.org.
Cite
@inproceedings{fitzgerald2018large,
title = {{Large-Scale QA-SRL Parsing}},
author = {FitzGerald, Nicholas and Michael, Julian and He, Luheng and Zettlemoyer, Luke},
booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2018},
doi = {10.18653/v1/p18-1191},
url = {https://doi.org/10.18653/v1/p18-1191}
}References
- Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (UCCA). In ACL 2013. find our card
- Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The Berkeley Framenet project. In ICCL 1998. find our card
- Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In 7th Linguistic Annotation Workshop and Interoperability with Discourse. find our card
- Valerio Basile, Johan Bos, Kilian Evang, and Noortje Venhuizen. 2012. Developing a large semantically annotated corpus. In LREC 2012. find
- Claire Bonial, Olga Babko-Malaya, Jinho D Choi, Jena Hwang, and Martha Palmer. 2010. Propbank annotation guidelines. find
- Ronan Collobert and Jason Weston. 2007. Fast semantic extraction using a novel neural network architecture. In ACL 2007. find
- Manjuan Duan, Ethan Hill, and Michael White. 2016. Generating disambiguating paraphrases for structurally ambiguous sentences. In 10th Linguistic Annotation Workshop. find
- Josef Ruppenhofer, Michael Ellsworth, Miriam RL Petruck, Christopher R Johnson, and Jan Scheffczyk. 2016. FrameNet II: Extended theory and practice. Institut für Deutsche Sprache, Bibliothek. find
- Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In EMNLP 2015. find
- Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In NIPS 2015. find
- Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In NIPS 2016. find our card
- Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In ACL 2018. find our card
- Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and whats next. In ACL 2017. find our card
- Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In EMNLP 2015. find our card
- Luheng He, Julian Michael, Mike Lewis, and Luke Zettlemoyer. 2016. Human-in-the-loop parsing. In EMNLP 2016. find
- Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR 2017. find
- Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In EMNLP 2016. find
- Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A Smith. 2017. Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold. arXiv preprint arXiv:1706.09528. ↗ our card
- Adam R Teichert, Adam Poliak, Benjamin Van Durme, and Matthew R Gormley. 2017. Semantic proto-role labeling. In AAAI 2017, pages 4459–4466. find
- Chenguang Wang, Alan Akbik, Laura Chiticariu, Yunyao Li, Fei Xia, and Anbang Xu. 2018. Crowd-in-the-loop: A hybrid approach for annotating semantic roles. In EMNLP 2017. find
- Aaron Steven White, Kyle Rawlins, and Benjamin Van Durme. 2017. The semantic proto-role linking model. In ACL 2017. find
- Bishan Yang and Tom Mitchell. 2017. A joint sequential and relational model for frame-semantic parsing. In EMNLP 2017, pages 1247–1256. find
- Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL 2015. find
- Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP 2017. find our card
- Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Berant. 2016. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436. ↗
- Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In ACL 2014, pages 55–60. find our card
- Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing question-answer meaning representations. In NAACL 2018. find our card
- Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics. find our card
- Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. TACL. find