Elamur Research⌕
☰
home / papers / 2014 / A New Entity Salience Task with Millions of Training Examples

A New Entity Salience Task with Millions of Training Examples

Summary

The paper presents entity salience as a task: given a document, decide which of its entities matter most. Many systems now reason over entities, yet importance is still usually judged with keyword methods. Rather than annotating by hand, the authors build training labels automatically from the annotated New York Times corpus, which pairs articles with abstracts written by staff. Both texts pass through an NLP pipeline (parsing, mention extraction, coreference, and Freebase entity resolution). A document entity is labeled salient if one of its proper-name mention heads matches an entity in the abstract.

Two experts annotated a sample, and their agreement with the induced labels came reasonably close to their agreement with each other. A logistic regression classifier was trained on positional, coreference-based mention-count, headline, and lexical features. It beats a first-sentence baseline by 34%. A PageRank-style centrality feature, built from entity co-occurrence statistics, adds only a small gain. The released dataset provides a large-scale resource for entity-centric retrieval, summarization, and advertising.

Key points

  • The dataset covers 110,639 NYT documents from 2003–2007, with 2,229,728 labeled entities. About 14% of those entities are salient. The 9,719 documents from 2007 form the test set.
  • Validation used 50 documents with 744 entities. Cohen's κ between the experts was 0.56 when scores 1–2 counted as salient, and 0.69 when only score 1 did. The experts' combined labels agreed with the NYT-derived labels at κ = 0.43 and 0.38 under the same two settings.
  • The model is L2-style regularized logistic regression trained with L-BFGS. It uses binary features hashed into 100,000 buckets and log-bucketed counts.
  • Results on the salient class:
    • The full feature set reaches F1 61.6, a 34% improvement over the baseline.
    • Coreference mention counts score F1 54.1, versus 44.4 for raw head-word counts.
    • Always predicting the majority class gives F1 23.9.
  • Weighted PageRank over the entity graph raises F1 to 62.0, which is significant at p ≤ 0.001 but small. The authors blame resolver errors, missing Freebase coverage, and overlap with the mention features. The labels and entity IDs were released at code.google.com/p/nyt-salience.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

25 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 21 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/e14-4040

@inproceedings{dunietz2014new,
  title = {{A New Entity Salience Task with Millions of Training Examples}},
  author = {Dunietz, Jesse and Gillick, Daniel},
  booktitle = {Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers},
  year = {2014},
  doi = {10.3115/v1/e14-4040},
  url = {https://doi.org/10.3115/v1/e14-4040}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Branimir Boguraev and Christopher Kennedy. 1997. Salience-based content characterisation of text documents. In Proceedings of the ACL, volume 97, pages 2–9. find
  2. Marco Cornolti, Paolo Ferragina, and Massimiliano Ciaramita. 2013. A framework for benchmarking entity-annotation systems. In Proceedings of the 22nd international conference on World Wide Web, pages 249–260. find
  3. Hal Daumé III and Daniel Marcu. 2005. Induction of word and phrase alignments for automatic document summarization. Computational Linguistics, 31(4):505–530. find
  4. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research (JAIR), 22(1):457–479. find
  5. Kuzman Ganchev and Mark Dredze. 2008. Small statistical models by random feature mixing. In Proceedings of the ACL08 HLT Workshop on Mobile Language Processing, pages 19–20. find
  6. Aria Haghighi and Dan Klein. 2009. Simple coreference resolution with rich syntactic and semantic features. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3-Volume 3, pages 1152–1161. Association for Computational Linguistics. find our card
  7. Anette Hulth. 2003. Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 216–223. find
  8. Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 423–430. find our card
  9. Ni Lao, Amarnag Subramanya, Fernando Pereira, and William W Cohen. 2012. Reading the web with learned syntactic-semantic inference rules. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1017–1026. Asso… find
  10. Edgar Meij, Wouter Weerkamp, and Maarten de Rijke. 2012. Adding semantics to microblog posts. In Proceedings of the fifth ACM international conference on Web search and data mining, pages 563– 572. ACM. find
  11. Rada Mihalcea and Andras Csomai. 2007. Wikify!: linking documents to encyclopedic knowledge. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pages 233–242. find
  12. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab. find our card
  13. Evan Sandhaus. 2008. The New York Times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752. find
  14. Takashi Tomokiyo and Matthew Hurst. 2003. A language model approach to keyphrase extraction. In Proceedings of the ACL 2003 workshop on Multiword expressions: analysis, acquisition and treatment-Volume 18, pages 33–40. find
  15. Marilyn A Walker, Aravind Krishna Joshi, and Ellen Friedman Prince. 1998. Centering theory in discourse. Oxford University Press. find
  16. Wenpu Xing and Ali Ghorbani. 2004. Weighted pagerank algorithm. In Communication Networks and Services Research, pages 305–314. IEEE. find
  17. Songhua Xu, Shaohui Yang, and Francis Chi-Moon Lau. 2010. Keyword extraction and headline generation using novel word features. In AAAI. find
  18. Chung Yong and Shou King Foo. 1999. A case study on inter-annotator agreement for word sense disambiguation. find

↑ back to top

This page in Markdown →