Probabilistic Coreference in Information Extraction

Summary

Most information extraction systems commit to a single answer when deciding which entity descriptions refer to the same thing. That leaves a downstream data-fusion component with no sense of how reliable the output is. Kehler instead treats template merging in SRI's FASTUS as a probabilistic problem. A "coreference set" is a group of templates that might corefer. The goal is a probability distribution over all ways of partitioning that set ("coreference configurations"), with known incompatibilities, such as clashing facility types, ruled out in advance.

Hand-coded training data is scarce and covers few large sets, so the approach has two stages:

A maximum entropy model (Berger et al., 1996) estimates coreference for individual template pairs.
Two strategies, "merging decision" and "evidential", combine the pairwise scores into a distribution over full configurations.

On held-out data, both strategies beat uniform and greedy baselines in cross-entropy, and the evidential method did best. The work matters because calibrated probabilities let fusion systems weigh IE output against other sources. The same machinery could also replace deterministic merging inside an IE system.

Key points

The setting is FASTUS applied to short military-style messages. In a worked example, four depot templates yield seven admissible configurations once type conflicts are excluded.
Pairwise coreference is learned with maximum entropy over binary features (e.g., identical slot values, phrase form, textual distance), trained via improved iterative scaling.
The pairwise models trained on merging-decision data reached a training cross-entropy of about 0.70, clearly lower than the evidential pairwise model. The paper attributes this to cleaner training data.
Evaluation cross-entropy (Table 2) followed the same ranking on all three training/test divisions:

Method Test Set 1 Test Set 2 Third division (more test data)

Uniform 2.12 1.76 2.01

Greedy 1.50 1.30 1.41

Merging decision 1.32 1.13 1.27

Evidential 1.10 0.89 1.00
Suggested future work includes richer context features (e.g., subject lines and tables, separate treatment of pronouns). Pruning and smoothing will be needed for longer texts with much larger coreference sets.

Method	Test Set 1	Test Set 2	Third division (more test data)
Uniform	2.12	1.76	2.01
Greedy	1.50	1.30	1.41
Merging decision	1.32	1.13	1.27
Evidential	1.10	0.89	1.00

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

22 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv cmp-lg/9706012

@inproceedings{kehler1997probabilistic,
  title = {{Probabilistic Coreference in Information Extraction}},
  author = {Kehler, Andrew},
  booktitle = {Second Conference on Empirical Methods in Natural Language Processing},
  year = {1997},
  eprint = {cmp-lg/9706012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/cmp-lg/9706012}
}

References

10 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Aone, Chinatsu and Scott William Bennett. 1995. Evaluating automated and manual acquisition of anaphora resolution strategies. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (ACL-95), pages 122-129, Cambridge, MA, June. find our card
Berger, Adam, Stephen A. Della Pietra, and Vincent J. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39-71. find our card
Connolly, Dennis, John D. Burger, and David S. Day. 1994. A machine learning approach to anaphoric reference. In Proceedings of the International Conference on New Methods in Language Processing (NeMLaP). find
Dagan, Ido and Alon Itai. 1990. Automatic acquisition of constraints for the resolution of anaphora references and syntactic ambiguities. In Proceedings of the 13th International Conference on Computational Linguistics (COLING-90), pages 330-332. find
Dagan, Ido, John Justenson, Shalom Lappin, Herbert Leass, and Amnon Ribak. 1995. Syntax and lexical statistics in anaphora resolution. Applied Artificial Intelligence, 9(6):633-644, Nov/Dec. find
Dempster, Arthur P. 1968. A generalization of Bayesian inference. Journal of the Royal Statistical Society, 30:205-247. find
Hobbs, Jerry R., Douglas E. Appelt, John Bear, David Israel, Megumi Kameyama, Mark Stickel, and Mabry Tyson. 1996. FASTUS: A cascaded finite-state transducer for extracting information from natural-language text. In Finite State Devices for Natural Language Processing. MIT Press, Cambridge, MA. find
Kennedy, Christopher and Branimir Boguraev. 1996a. Anaphora for everyone: Pronominal anaphora resolution without a parser. In Proceedings of the 16th International Conference on Computational Linguistics (COLING-96). find our card
Kennedy, Christopher and Branimir Boguraev. 1996b. Anaphora in a wider context: Tracking discourse referents. In Proceedings of the 12th European Conference on Artificial Intelligence (ECAI-96). find
Lappin, Shalom and Herbert Leass. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4):535-561. find our card

↑ back to top

📄 Probabilistic Coreference in Information Extraction