Elamur Research⌕
☰
home / papers / 2014 / Modeling Biological Processes for Reading Comprehension

Modeling Biological Processes for Reading Comprehension

Summary

Most machine reading research harvests facts from huge, redundant web collections. This paper instead targets micro-reading: answering questions about one paragraph, where the answer depends on how events and entities in a described process relate to each other. The authors build ProcessBank, a set of 200 biology textbook paragraphs. Each paragraph is annotated with a process graph and paired with two-option multiple-choice questions written by a biologist. Their system works in two stages. First, a supervised predictor extracts event triggers, arguments, semantic roles and event-event relations across sentences, using structural constraints during inference. Second, a rule-based procedure converts each question plus candidate answer into a graph query and runs it over the predicted structure, which acts as a tiny knowledge base. Reasoning over these predicted structures answers questions substantially more accurately than baselines built on shallower signals such as text or syntactic proximity. This shows that explicit structure helps on comprehension questions that lexical matching cannot solve.

Key points

  • ProcessBank data:
    • 200 paragraphs from Campbell and Reece's Biology, extending the 148 identified by Scaria et al. (2013).
    • 585 questions in total.
    • A second annotator checked 326 of them and agreed on the answer in 98.1% of cases.
  • Process graph:
    • Triggers are marked T-YES or T-NO, depending on whether the event happens.
    • Arguments receive roles: AGENT, THEME, SOURCE, DESTINATION, LOCATION, RESULT, OTHER.
    • Events connect through eight relations: CAUSE, ENABLE and PREVENT, their -OR (disjunctive) variants, SUPER (sub-event) and SAME (event coreference).
  • Annotation:
    • Three biologists annotated structures in the brat tool, with one annotating and another validating each paragraph.
    • A paragraph averages 7.0 triggers, 11.3 arguments and 7.9 relations.
    • Questions were written independently of the structure annotation.
  • Relation to Scaria et al. (2013): temporal relations are dropped, on the argument that causal links already imply the ordering that matters.
  • Error analysis: the authors sampled 20 errors from the system using gold structures (GOLD) and 20 from the system using predicted structures (ProRead). Structure-prediction mistakes cause over half of ProRead's errors. GOLD's errors are spread across alignment, missing annotation, entity coreference, lexical variability and query-pattern coverage.

Interesting ideas

  • I1. Balanced distractors: both answer options are built to overlap the paragraph lexically to a similar degree, and they reuse the paragraph's exact wording. This blocks word-matching shortcuts and isolates genuine comprehension, a useful recipe for any QA benchmark meant to test reasoning.
  • I2. Answering as querying: each question-answer pair becomes a query over a structure extracted from the document. This splits the problem into extraction and querying, so you can measure how much error each part contributes, for example by running the same queries on gold versus predicted structures.
  • I3. Conjunction versus disjunction in relation labels: having -OR versions of the causal relations lets the graph express that an event needs several causes, or leads to only one of several outcomes. This is a cheap way to add logical nuance to an event-relation schema.
  • I4. Constraint-aware inference: joint prediction enforces structural consistency, for instance forbidding mutual causation between two events. Such global constraints can help any graph-extraction task where labels on different edges depend on each other.
  • I5. Separate annotators for questions and structures: because the question writer never saw the structure annotation, the evaluation is not biased toward the schema. It also exposes gaps in the annotation, which is a good practice when building paired resources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

93 papers of our library cite it (of the 140884 whose bibliographies we read); 21 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1159

@inproceedings{berant2014modeling,
  title = {{Modeling Biological Processes for Reading Comprehension}},
  author = {Berant, Jonathan and Srikumar, Vivek and Chen, Pei-Chun and Linden, Abby Vander and Harding, Brittany and Huang, Brad and Clark, Peter and Manning, Christopher D.},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1159},
  url = {https://doi.org/10.3115/v1/d14-1159}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of EMNLP. find
  2. Neil Campbell and Jane Reece. 2005. Biology. Benjamin Cummings. find
  3. Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka Jr., and Tom M. Mitchell. 2010. Toward an architecture for never-ending language learning. In Proceedings of AAAI. find
  4. Nathanael Chambers and Daniel Jurafsky. 2008. Jointly combining implicit constraints improves temporal ordering. In Proceedings of EMNLP. find
  5. Wen-Chi Chou, Richard Tzong-Han Tsai, Ying-Shan Su, Wei Ku, Ting-Yi Sung, and Wen-Lian Hsu. 2006. A semi-automatic method for annotating a biomedical proposition bank. In Proceedings of the Workshop on Frontiers in Linguistically Annotated Corpora 2006, July. find
  6. James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world’s response. In Proceedings of CoNLL. find
  7. James Clarke, Vivek Srikumar, Mark Sammons, and Dan Roth. 2012. An NLP Curator (or: How I Learned to Stop Worrying and Love NLP Pipelines). In Proceedings of LREC. find
  8. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of ACL. find our card
  9. Quang Do, Wei Lu, and Dan Roth. 2012. Joint inference for event timeline construction. In Proceedings of EMNLP-CoNLL. find
  10. George R. Doddington, Alexis Mitchell, Mark A. Przybocki, Lance A. Ramshaw, Stephanie Strassel, and Ralph M. Weischedel. 2004. The Automatic Content Extraction (ACE) Program-Tasks, Data, and Evaluation. In Proceedings of LREC. find our card
  11. Oren Etzioni, Michele Banko, and Michael J. Cafarella. 2006. Machine reading. In Proceedings of AAAI. find
  12. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of EMNLP. find
  13. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-Driven Learning for Open Question Answering. In Proceedings of ACL. find our card
  14. Christiane Fellbaum, editor. 1998. WordNet: An electronic lexical database. MIT Press. find
  15. Lynette Hirschman, Marc Light, Eric Breck, and John D. Burger. 1999. Deep read: A reading comprehension system. In Proceedings of ACL. find
  16. Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Junichi Tsujii. 2009. Overview of BioNLP 09 shared task on event extraction. In Proceedings of BioNLP. find
  17. Jin-Dong Kim, Sampo Pyysalo, Tomoko Ohta, Robert Bossy, and Junichi Tsujii. 2011. Overview of BioNLP shared task 2011. In Proceedings of BioNLP. find
  18. Jayant Krishnamurthy and Thomas Kollar. 2013. Jointly learning to parse and perceive: Connecting natural language to the physical world. TACL, 1:193–206. find
  19. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of ACL. find
  20. Beth Levin. 1993. English verb classes and alternations: A preliminary investigation. University of Chicago Press. find
  21. Catherine Macleod, Ralph Grishman, Adam Meyers, Leslie Barrett, and Ruth Reeves. 1998. Nomlex: A lexicon of nominalizations. In Proceedings of EU-RALEX. find
  22. Thomas L. Magnanti and Laurence A. Wolsey. 1995. Optimal trees. Handbooks in operations research and management science, 7:503–615. find
  23. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of ACL: System Demonstrations. find our card
  24. André L. Martins, Noah A. Smith, and Eric P. Xing. 2009. Concise integer linear programming formulations for dependency parsing. In Proceedings of ACL/IJCNLP. find
  25. David McClosky and Christopher D. Manning. 2012. Learning constraints for consistent timeline extraction. In Proceedings of EMNLP-CoNLL. find
  26. Dan Moldovan, Christine Clark, Sanda Harabagiu, and Steve Maiorano. 2003. Cogex: A logic prover for question answering. In Proceedings of NAACL-HLT. find
  27. Martha Palmer, Paul Kingsbury, and Daniel Gildea. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. find our card
  28. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2). find our card
  29. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of EMNLP. find our card
  30. Sebastian Riedel and Andrew McCallum. 2011. Fast and robust joint models for biomedical event extraction. In Proceedings of EMNLP. find
  31. Dan Roth and Wen-tau Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Proceedings of CoNLL. find our card
  32. Josef Ruppenhofer, Michael Ellsworth, Miriam RL Petruck, Christopher R. Johnson, and Jan Scheffczyk. 2006. FrameNet II: Extended theory and practice. Berkeley FrameNet Release, 1. find
  33. Aju Thalappillil Scaria, Jonathan Berant, Mengqiu Wang, Peter Clark, Justin Lewis, Brittany Harding, and Christopher D. Manning. 2013. Learning biological processes with global constraints. In Proceedings of EMNLP. find
  34. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. Brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the demonstrations at EACL. find our card
  35. Yuk Wah Wong and Raymond J. Mooney. 2006. Learning for semantic parsing with statistical machine translation. In Proceedings of HLT-NAACL. find
  36. Katsumasa Yoshikawa, Sebastian Riedel, Masayuki Asahara, and Yuji Matsumoto. 2009. Jointly identifying temporal relations with Markov logic. In Proceedings of ACL/IJCNLP. find
  37. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of UAI. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/berant-2014-modeling-biological-processes-reading-comprehension/