Elamur Research⌕
☰
home / papers / 2009 / A Unified Model of Phrasal and Sentential Evidence for Information Extraction

A Unified Model of Phrasal and Sentential Evidence for Information Extraction

Paper · 2009 · Siddharth Patwardhan and Ellen Riloff

Summary

Event-oriented information extraction systems usually judge a candidate noun phrase by its immediate surroundings. This misses role fillers that sit in clauses with no event word, such as "people died" or "bridges were destroyed," which could follow many kinds of incidents. Patwardhan and Riloff propose GLACIER, a probabilistic model for noun phrase extraction. It treats the decision as a joint probability: the sentence describes a relevant event, and the phrase is a plausible filler for a given role. The product rule splits this into two components:

  • a sentential event recognizer, built with Naïve Bayes or SVM classifiers;
  • a plausible role-filler recognizer, a Naïve Bayes classifier over local lexical, semantic and syntactic features, conditioned on the sentence being an event sentence.

Neither component makes a hard keep-or-drop decision about a sentence. The authors evaluate on MUC-4 (terrorism) and ProMed (disease outbreaks) against pattern-based and classifier baselines. GLACIER wins clearly on most roles, with gains in recall and in precision. The work shows that sentence-level context can rescue facts that local evidence alone treats as too vague.

Key points

  • MUC-4, NB/SVM variant (threshold 0.40):
    • It beats all three baselines (AutoSlog-TS, a Naïve Bayes baseline, Sem Affinity) on PerpInd, Victim and Weapon.
    • PerpInd F-score rose from 0.43 (Sem Affinity) to 0.54.
    • Victim and Weapon each gained about 5% F-score over the best baseline.
  • Weak spot: PerpOrg results were worse than the baselines. Responsible organizations tend to be named later in a document, away from the event description, so their sentences receive low event probabilities.
  • ProMed:
    • The NB/NB variant worked better than NB/SVM in this domain.
    • Disease F-score went from 0.43 to 0.49, mainly through higher precision (up to 0.41).
    • Victim F-score tied the best baseline at 0.44, with 5% more recall and 3% less precision.
  • Implementation: Weka for Naïve Bayes and SVMLight for the SVM. Extraction thresholds were 0.90 for NB/NB, and 0.50 or 0.40 for NB/SVM. Training required event sentences to be annotated.
  • Shared component: One sentential event recognizer serves all roles, while each role gets its own role-filler recognizer.

Interesting ideas

  • I1. Product-rule factorization instead of a pipeline. Multiplying a context-level probability by a local probability conditioned on that context replaces a filter-then-extract cascade. Strong local evidence can still win in a weak sentence, and weak local evidence can pass in a strong sentence. This suits any task where a span's relevance depends on a larger unit, such as relation extraction or document-level tagging.
  • I2. Conditioning the local classifier on the context class. The role-filler model is trained to assume it is looking at an event sentence. It can therefore accept generic phrases that would be too risky in isolation, because the sentence component supplies the topical evidence.
  • I3. Using a better-calibrated classifier where probabilities matter. Naïve Bayes pushes its estimates toward 0 or 1, so the authors tried an SVM for the sentence component. Score calibration deserves attention whenever classifier outputs are combined rather than just used as labels.
  • I4. Tuning the extraction threshold per configuration. The SVM-based variant needed a threshold below 0.5 to recover recall. Reporting results at several thresholds is a cheap way to avoid underselling a combined model.
  • I5. Reusing one context model across many targets. A single sentence-relevance model feeds every role-specific extractor, which reduces annotation and training cost when a schema has many slots.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of UtahSalt Lake City · United States29 works · 5,781 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 109 works

Cite

Cite as Siddharth Patwardhan and Ellen Riloff. 2009. A Unified Model of Phrasal and Sentential Evidence for Information Extraction. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{patwardhan2009unified,
  title = {{A Unified Model of Phrasal and Sentential Evidence for Information Extraction}},
  author = {Siddharth Patwardhan and Ellen Riloff},
  booktitle = {Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  doi = {10.3115/1699510.1699530},
  url = {https://doi.org/10.3115/1699510.1699530}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Bunescu and R. Mooney. 2004. Collective Information Extraction with Relational Markov Networks. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 438–445, Barcelona, Spain, July. find
  2. M. Califf and R. Mooney. 2003. Bottom-Up Relational Learning of Pattern Matching Rules for Information Extraction. Journal of Machine Learning Research, 4:177–210, December. find
  3. H. Chieu, H. Ng, and Y. Lee. 2003. Closing the Gap: Learning-Based Information Extraction Rivaling Knowledge-Engineering Methods. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 216–223, Sapporo, Japan, July. find
  4. P. Domingos and M. Pazzani. 1996. Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier. In Proceedings of the Thirteenth International Conference on Machine Learning, pages 105–112, Bari, Italy, July. find
  5. J. Finkel, T. Grenager, and C. Manning. 2005. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 363–370, Ann Arbor, MI, June.
  6. D. Freitag and A. McCallum. 2000. Information Extraction with HMM Structures Learned by Stochastic Optimization. In Proceedings of the Seventeenth National Conference on Artificial Intelligence, pages 584–589, Austin, TX, August. find
  7. D. Freitag. 1998. Toward General-Purpose Learning for Information Extraction. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, pages 404–408, Montreal, Quebec, August. find
  8. H. Ji and R. Grishman. 2008. Refining Event Extraction through Cross-Document Inference. In Proceedings of ACL-08: HLT, pages 254–262, Columbus, OH, June.
  9. T. Joachims. 1998. Text Categorization with Support Vector Machines: Learning with Many Relevant Features. In Proceedings of the Tenth European Conference on Machine Learning, pages 137–142, April.
  10. C. Manning, P. Raghavan, and H Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York, NY. find
  11. M. Maslennikov and T. Chua. 2007. A Multi-resolution Framework for Information Extraction from Free Text. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 592–599, Prague, Czech Republic, June. find
  12. S. Patwardhan and E. Riloff. 2007. Effective Information Extraction with Semantic Affinity Patterns and Relevant Regions. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 717–727, Prague, Czech Republic… find
  13. W. Phillips and E. Riloff. 2007. Exploiting Role-Identifying Nouns and Expressions for Information Extraction. In Proceedings of International Conference on Recent Advances in Natural Language Processing, pages 165–172, Borovets, Bulgaria, September. find
  14. E. Riloff and W. Phillips. 2004. An Introduction to the Sundance and AutoSlog Systems. Technical Report UUCS-04-015, School of Computing, University of Utah. find
  15. E. Riloff. 1996. Automatically Generating Extraction Patterns from Untagged Text. In Proceedings of the Thirteenth National Conference on Articial Intelligence, pages 1044–1049, Portland, OR, August. find
  16. S. Soderland, D. Fisher, J. Aseltine, and W. Lehnert. 1995. CRYSTAL: Inducing a Conceptual Dictionary. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pages 1314–1319, Montreal, Canada, August. find
  17. B. Sundheim. 1992. Overview of the Fourth Message Understanding Evaluation and Conference. In Proceedings of the Fourth Message Understanding Conference (MUC-4), pages 3–21, McLean, VA, June. find
  18. V. Vapnik. 1995. The Nature of Statistical Learning Theory. Springer, New York, NY. find
  19. I. Witten and E. Frank. 2005. Data Mining - Practical Machine Learning Tools and Techniques. Morgan– Kaufmann, San Francisco, CA. find
  20. R. Yangarber, R. Grishman, P. Tapanainen, and S. Huttunen. 2000. Automatic Acquisition of Domain Knowledge for Information Extraction. In Proceedings of the 18th International Conference on Computational Linguistics, pages 940–946, Saarbrücken, Germany, August.
  21. B. Zadrozny and C. Elkan. 2001. Obtaining Calibrated Probability Estimates from Decision Trees and Naı̈ve Bayesian Classiers. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 609–616, Williamstown, MA, June. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/patwardhan-2009-unified-model-phrasal-sentential-evidence/