Elamur Research⌕
☰
home / papers / 2002 / Learning Question Classifiers

Learning Question Classifiers

Summary

The paper treats question classification as a learning problem in its own right. The goal is to predict the semantic type of the answer a factual question expects, so that a downstream question answering system can discard unsuitable candidates and choose type-specific search strategies. Earlier TREC systems mostly used hand-written rules with a coarse inventory of roughly 20 types or fewer. Li and Roth instead propose a two-level taxonomy with 6 coarse and 50 fine answer classes. They train a pair of chained classifiers built on the SNoW architecture with Winnow. The coarse prediction narrows the candidate label set for the fine stage. Because many questions are genuinely ambiguous, the system may return several labels per question. With 5,500 training questions and the 500 TREC 10 questions as a test set, the full feature set (including semantically related word lists) scores 98.8% on coarse and 95% on fine classes when up to five labels are allowed. The results show that fine-grained answer typing can be learned accurately, and that semantic features matter much more than surface ones.

Key points

  • Taxonomy: six coarse classes (ABBREVIATION, ENTITY, DESCRIPTION, HUMAN, LOCATION, NUMERIC VALUE) split into 50 non-overlapping fine classes.
  • Data: training on 5,500 labelled questions; evaluation on the 500 TREC 10 questions.
  • Features: added cumulatively in the order Word, Pos, Chunk, NE, Head, RelWord.
    • Fine-class accuracy with up to five labels rises from 86.0% (words only) to 95.0% (with RelWord).
    • The jump from RelWord is much larger for fine classes than for coarse ones.
  • Single-label setting: accuracy with one label is 91.0% for coarse and 84.2% for fine classes.
    • On average the coarse classifier outputs 1.54 labels and the fine classifier 2.05.
  • Flat vs. hierarchical: a flat classifier over all 50 fine classes performs about as well as the hierarchical one. The authors therefore find no real benefit from the coarse layer.

Interesting ideas

  • I1. Multiple labels for ambiguous inputs: the classifier returns a small set of plausible labels, obtained by thresholding probabilities derived from its activations. Downstream components keep every reasonable reading at little cost, which suits any pipeline where an early hard decision would spread errors.
  • I2. Label filtering between stages: the coarse stage's surviving classes are expanded into the confusion set for the fine stage. This is a cheap way to impose a label hierarchy on any multi-class problem.
  • I3. Feature types instead of hand-listed features: a handful of feature types are defined and then expanded from the data into a very large feature space. This avoids writing rules by hand and makes retraining on a new taxonomy fast, which is helpful when adapting to a new language or domain.
  • I4. Semantic word-class features: lists of semantically related words give the biggest gain, especially for fine classes. Lexical-semantic resources can therefore matter more than extra syntactic layers.
  • I5. Checking whether a hierarchy helps: the authors measure normalised pairwise confusion between fine classes and see that errors do not cluster inside coarse classes. This diagnostic is worth running before assuming a hierarchy will help a classifier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

411 papers of our library cite it (of the 140884 whose bibliographies we read); 27 of them have a page here, 384 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{li2002learning,
  title = {{Learning Question Classifiers}},
  author = {Li, Xin and Roth, Dan},
  booktitle = {COLING 2002: The 19th International Conference on Computational Linguistics},
  year = {2002},
  doi = {10.3115/1072228.1072378},
  url = {https://doi.org/10.3115/1072228.1072378}
}

References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. P. Abney. 1991. Parsing by chunks. In S. P. Abney R. C. Berwick and C. Tenny, editors, Principle-based parsing: Computation and Psycholinguistics, pages 257–278. Kluwer, Dordrecht. find
  2. A. Carlson, C. Cumby, J. Rosen, and D. Roth. 1999. The SNoW learning architecture. Technical Report UIUCDCS-R-99-2101, UIUC Computer Science Department, May. find
  3. C. Cumby and D. Roth. 2000. Relational representations that facilitate learning. In Proc. of the International Conference on the Principles of Knowledge Representation and Reasoning, pages 425–434. find
  4. Y. Even-Zohar and D. Roth. 2001. A sequential model for multi class classification. In EMNLP-2001, the SIGDAT Conference on Empirical Methods in Natural Language Processing, pages 10–19. find
  5. A. R. Golding and D. Roth. 1999. A Winnow based approach to context-sensitive spelling correction. Machine Learning, 34(1-3):107–130. find
  6. S. Harabagiu, D. Moldovan, M. Pasca, R. Mihalcea, M. Surdeanu, R. Bunescu, R. Girju, V. Rus, and P. Morarescu. 2001. Falcon: Boosting knowledge for answer engines. In Proceedings of the 9th Text Retrieval Conference, NIST. find
  7. U. Hermjakob. 2001. Parsing and question classification for question answering. In ACL-2001 Workshop on Open-Domain Question Answering. find
  8. L. Hirschman, M. Light, E. Breck, and J. Burger. 1999. Deep read: A reading comprehension system. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics. find
  9. E. Hovy, L. Gerber, U. Hermjakob, C. Lin, and D. Ravichandran. 2001. Toward semantics-based answer pinpointing. In Proceedings of the DARPA Human Language Technology conference (HLT). San Diego, CA. find our card
  10. A. Ittycheriah, M. Franz, W-J Zhu, A. Ratnaparkhi, and R.J. Mammone. 2001. IBM’s statistical question answering system. In Proceedings of the 9th Text Retrieval Conference, NIST. find
  11. W. G. Lehnert. 1986. A conceptual theory of question answering. In B. J. Grosz, K. Sparck Jones, and B. L. Webber, editors, Natural Language Processing, pages 651–657. Kaufmann, Los Altos, CA. find
  12. X. Li and D. Roth. 2001. Exploring evidence for shallow parsing. In Proc. of the Annual Conference on Computational Natural Language Learning. find
  13. M. Light, G. Mann, E. Riloff, and E. Breck. 2001. Analyses for Elucidating Current Question Answering Technology. Journal for Natural Language Engineering. forthcoming. find
  14. V. Punyakanok and D. Roth. 2001. The use of classifiers in sequential inference. In NIPS-13; The 2000 Conference on Advances in Neural Information Processing Systems, pages 995–1001. MIT Press. find
  15. D. Roth and W. Yih. 2001. Relational learning via propositional algorithms: An information extraction case study. In Proc. of the International Joint Conference on Artificial Intelligence, pages 1257–1263. find
  16. D. Roth, G. Kao, X. Li, R. Nagarajan, V. Punyakanok, N. Rizzolo, W. Yih, C. O. Alm, and L. G. Moran. 2002. Learning components for a question answering system. In TREC-2001. find
  17. D. Roth. 1998. Learning to resolve natural language ambiguities: A unified approach. In Proc. of the American Association of Artificial Intelligence, pages 806– 813. find
  18. A. Singhal, S. Abney, M. Bacchiani, M. Collins, D. Hindle, and F. Pereira. 2000. AT&T at TREC-8. In Proceedings of the 8th Text Retrieval Conference, NIST. find
  19. E. Voorhees. 2000. Overview of the TREC-9 question answering track. In The Ninth Text Retrieval Conference (TREC-9), pages 71–80. NIST SP 500-249. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/