Elamur Research⌕
☰
home / papers / 2012 / Question Classification and Answering from Procedural Text in English

Question Classification and Answering from Procedural Text in English

Paper · 2012 · Somnath Banerjee and Sivaji Bandyopadhyay

Summary

The paper looks at how-to questions, which ask how to carry out a procedure such as a recipe, a repair or a medical routine. Most earlier work on question classification targeted factoid questions, and research on procedural text had focused on French. The authors gathered 6,230 English questions from Yahoo Answers spanning four subject areas (medicine, electronics, home and maintenance, and cooking recipes) and retained 6,081 of them as valid. They split these 70/30 into a pattern-discovery set and a test set. After tagging the questions with the Stanford POS and NER tools, they defined hand-crafted surface patterns and grouped them into seven classes. Each class points to a different part of a procedure: prerequisites, direction, special information, justification, advice, warning and simple instruction. A shallow parser builds a small query parse tree and matches it against the patterns. The predicted class then selects the matching tagged section of a procedure document. On 1,824 test questions the system reached 95.08% precision, 86.95% recall and 90.84 F-measure. This suggests that a compact, procedure-aware taxonomy handles how-to questions well.

Key points

  • Data: 6,081 valid procedural questions from Yahoo Answers, split into 4,257 for finding patterns and 1,824 for testing. Manual filtering took about 32 hours.
  • Seven classes are used: PA, DA, SpIA, JA, AA, WA and SIA. They are defined by about a dozen rule patterns of the form <WH><slot><X>, with slots such as ITEM, GOAL, STEP, and verbs of advice, preference, warning or prevention. The abstract reports annotator agreement of 94.6% kappa.
  • The document side has 200 procedures, 50 per domain, from the BBC recipe site, eHow, Home Repair and Health.com. Their instructions are labelled as simple, advice or warning. A typical procedure has about 9 instructional compounds of roughly 13 tokens each.
  • Results: 1,668 of the test questions received a class, and 1,586 of those were correct (P 95.08%, R 86.95%, F 90.84). Per-class F ranges from about 85 to 95 across the four domains.
  • The authors find that SpIA patterns depend heavily on the domain and require domain-specific knowledge.

Interesting ideas

  • I1. The question taxonomy mirrors the anatomy of a procedure (prerequisites, steps, justification, advice, warnings), so the question class directly names the part of the document that holds the answer.
  • I2. Instructions inside procedural documents are split into simple, advice and warning types, which treats a text as a structured plan rather than a flat list of sentences.
  • I3. Small verb lexicons (warning, prevention, preference/advice verbs) act as slots in the question patterns, which carries a semantic signal into an otherwise surface-level method.
  • I4. The classifier may decline to label a question, so precision and recall are reported separately and the gap between them shows how often the system abstains.
  • I5. Results are broken down by domain and class, and this exposed one class (SpIA) whose patterns do not transfer across domains.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): A how-to reader could map question types to specific slots of a procedure in the world of the text, such as ingredients, ordered steps or cautions, rather than searching free text.
  • I2 (world): The world builder could store procedures as ordered steps with attached advice and warning nodes, so step-order and "what to avoid" questions can be answered from the structure.
  • I3 (frames): Verb classes from VerbNet could tag frames as advice, warning or prohibition speech acts, giving prag an explainable cue for directive and cautionary acts.
  • I4 (learn): Evaluation of Mova readers could count abstentions apart from wrong answers, which rewards saying "unknown" over guessing.
  • I5 (learn): Per-domain breakdowns in test reports would reveal which question types depend on domain knowledge and need compiled ontology support rather than more rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Somnath Banerjee and Sivaji Bandyopadhyay. 2012. Question Classification and Answering from Procedural Text in English. Proceedings of the Workshop on Question Answering for Complex Domains.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{banerjee2012question,
  title = {{Question Classification and Answering from Procedural Text in English}},
  author = {Somnath Banerjee and Sivaji Bandyopadhyay},
  booktitle = {Proceedings of the Workshop on Question Answering for Complex Domains},
  year = {2012},
  url = {https://aclanthology.org/W12-6002/}
}

References

32 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aouladomar, F. (2005). A preliminary analysis of the discursive and rhetorical structure of procedural texts. In Symposium on the Exploration and Modeling of Meaning. find
  2. Aouladomar, F. and Saint-Dizier, P. (2005). An Exploration of the Diversity of Natural Argumentation in Instructional Texts. In 5th International Workshop on ComputationalModels of Natural Argument, IJCAI, Edinburgh. find
  3. Aouladomar, F. and Saint-Dizier, P. (2005). Towards Answering Procedural Questions. In Proceedings of Workshop KRAQ05, IJCAI05, Edinburgh. find
  4. Moschitti, A., Quarteroni, S., Basili, R. and Manandhar, S. (2007). Exploiting syntactic and shallow semantic kernels for question/answer classification. In ACL, pages 776–783, 2007. find
  5. Adam, C., Delpech, E. and Saint-Dizier, P. (2008). Identifying and Expanding Titles in Web Texts. In Proceedings of ACM. find
  6. Pechsiri, C., Sroison, P. and Janviriyasopa, U. (2008). Know-why extraction from textual data. In Proceedings of KRAQ. find
  7. Zhang, D. and Lee, W. S. and Lee, S. (2003). Question classification using support vector machines. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR ’03, pages 26–32, New York, NY, USA,ACM. find
  8. Delpech, E. and Saint-Dizier, P. (2008). Investigating the structure of procedural texts for answering how-to questions. In Proceedings of LREC. find
  9. Voorhees, E. M. (2003). Overview of TREC 2003. In Proceedings of TREC. find
  10. Greimas, A. (1983). La Soupe au Pistou ou la Conservation d’unObjet de Valeur, In Du sens II, Seuil, Paris. find
  11. Hripcsak, G. and Rothschild, A. (2005). Agreement, the F-measure and reliability in information retrieval: In Journal of the American Medical Informatics Association, pages 296-298. find
  12. Finkel, J. R., Grenager, T and Manning, C. (2005). Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pages 363-370.
  13. Toutanova, K., Klein, D., Manning, C. and Singer, Y. (2003). Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. In Proceedings of HLT-NAACL 2003, pages 252-259.
  14. Kosseim, L. and Lapalme, G. (2000). Choosing Rhetorical Structuresto Plan InstructionalTexts, Computational Intelligence, Blackwell, Boston. find
  15. Fontan, L. and Saint-Dizier, P. (2008). Creating and Querying a Domain dependent Know-How Knowledge Base of Advices and Warnings. In Proceedings of KRAQ. find
  16. Yin, L. (2006). A two-stage approach to retrieving answers for how-to questions. In Proceedings of EACL (Student Session). find
  17. Fontan, L. and Saint-Dizier, P. (2008). Analyzing Argumentative Structures in Procedural Texts. GoTAL 2008, pages 366-370 find
  18. Fontan, L. and Saint-Dizier, P. (2008). Constructing a know-how repository of advices and warnings from procedural texts. ACM Symposium on Document Engineering 2008, pages 249-252 Cai, L., Zhou, G., Liu, L. and Zhao, J. () Large-Scale Question Classification in cQA by Leveraging Wikipedia Semantic Knowledge. Mortara Garavelli, B., Tipologia dei Testi, In (G. find
  19. Hodus et al., 1988: lexicon der romanistischen Linguistik, vol. IV, Tubingen, Niemeyer). find
  20. Simmons, R. F. (1965). Answering English questions by computer: a survey. Comm. ACM, 8(1):53–70. find
  21. S. Verberne, L. Boves, N. Oostdijk, and P. Coppen. 2007. Evaluating discourse-based answer extraction for why-question answering. In Proceedings of SIGIR, pages 735–737. find
  22. Saint-Dizier, P. (2008). Some Challenges of Advanced Question-Answering: an Experiment with How-to Questions. In Proceedings of PACLIC 2008, pages 65-73 find
  23. Nguyen, T., Nguyen, L. and Shimazu, A. (2008). Using semi-supervised learning for question classification. Journal Natural Language Processing, 15(1):3–21. find
  24. Zhang, Z., Uren, V., Ciravegna, F. (2010). Position paper: A comprehensive solution to procedural knowledge acquisition using information extraction. In Proceedings of KDIR2010, Valencia. find
  25. Li, X. and Roth, D. (2004). Learning question classifiers: The role of semantic information. In Proceedings of International Conference on Computational Linguistics (COLING), pages 556– 562. Huang, Z., Thint, M. and Qin. ↗
  26. Z. (2008). Question classification using head words and their hypernyms. In Proceedings of EMNLP, pages 927–936. find
  27. Silva, J., Coheur, L., Mendes, A. and Wichert, A. (2011). From symbolic to sub-symbolic information in question classification. Artifcial Intelligence Review, 35(2):137–154. ↗
  28. Merkel, A. and Klakow, D. (2007). Improved methods of language model based question classification. In Proceedings of Interspeech Conference. find
  29. Blunsom, P., Kocik, K. and Curran, J. R. (2006). Question classification with log-linear models. In Proceedings of SIGIR ’06, pages 615–616, NY, USA, ACM. find
  30. De Rijke, M. (2005). Question Answering: What’s Next?, In Sixth International Workshop on Computational Semantics, Tilburg. find
  31. Yin, L. (2004). Topic Analysis and Answering Procedural Questions, Information Technology Research Institute Technical Report Series, ITRI-04-14, University of Brighton, UK. find
  32. Loni, B., Tulder,G., Wiggers, P., Loog, M. and Tax, D. (2011).Question classification with weighted combination of lexical, syntactical and semantic features. In Proceedings of the 15th international conference of Text, Dialog and Speech. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/banerjee-2012-question-classification-answering-procedural-text/