Elamur Research⌕
☰
home / papers / 2017 / Automatically Labeled Data Generation for Large Scale Event Extraction

Automatically Labeled Data Generation for Large Scale Event Extraction

Paper · 2017 · Yubo Chen, Shulin Liu, Xiang Zhang et al.

Summary

Supervised event extraction (EE) systems are usually trained on ACE 2005. That corpus is expensive to annotate and small: it defines only 33 event types over 599 English documents, and most types have few examples. The authors adapt distant supervision, previously used for relation extraction, to EE.

Two obstacles make the direct transfer hard. Knowledge bases store events as abstract nodes with no trigger words. Also, requiring every argument of an event to appear in one sentence finds almost no matches.

Their pipeline treats Freebase CVTs as event instances and ranks each type's arguments to keep only a few key ones. It then labels Wikipedia sentences that contain those key arguments and takes characteristic verbs as candidate triggers. FrameNet is used to prune noisy triggers and add nominal ones. A Soft Distant Supervision step then builds the final corpus.

Manual and automatic checks show the generated data is competitive with human annotation, and adding it to ACE data improves extractors. A DMCNN with multi-instance learning serves as a baseline. The work offers a cheap route to EE across many event types.

Key points

  • Resources used: Freebase (dump of April 2015, about 1885 CVTs and about 14M instances), FrameNet (over 1,000 frames and 10,000 LUs), and the January 2016 Wikipedia (6.3M articles). From Freebase, 21 event types with about 3.8M instances were kept.
  • Matching all arguments of an event in a single sentence works for only 0.02% of instances, which motivated the key-argument strategy.
  • The number of key arguments k strongly affects corpus size. For people.marriage, k=2 gave 25,797 labeled sentences, while k=3 gave only 534. With k=1, precision suffered.
  • Compared with ACE+TR, adding FrameNet to trigger detection improved trigger classification by 1.0 and argument classification by 1.7.
  • In human evaluation of the top 100/300/500 extracted events, DMCNNs+MIL averaged 67.4 precision for event classification and 46.6 for argument classification. Plain DMCNNs averaged 55.3 and 40.3. The labeled data was released publicly on GitHub.

Interesting ideas

  • I1. Key arguments instead of all arguments: each argument role gets a score combining how often it appears within an event type with how specific it is to that type, similar in spirit to TF-IDF. Only the top roles are used for matching. This trades a little precision for far more coverage, which suits any distant-labeling task with complex, multi-slot records.
  • I2. Discovering triggers from aligned sentences: verbs that are frequent in sentences labeled for one event type but rare across other types become candidate triggers. The same idea could build keyword lexicons for new domains or languages, such as Ukrainian news, without manual annotation.
  • I3. Using a lexical resource as a filter and an expander: FrameNet both removes noisy verbal triggers and adds nominal triggers. This shows how a curated linguistic resource can correct a statistically mined lexicon in both directions.
  • I4. Combining two kinds of evaluation: held-out comparison against the knowledge base is backed by manual precision at top-n. The manual check catches false negatives caused by the KB's incompleteness, which makes it a sensible protocol for any weakly supervised dataset.
  • I5. Multi-instance learning for noisy labels: treating groups of distantly labeled sentences as bags makes the model tolerate wrong labels. This is a reusable recipe whenever training data comes from automatic alignment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China68 works · 22,753 citations🔬Institute of AutomationBeijing · China20 works · 13,875 citations🎓University of Chinese Academy of SciencesBeijing · China32 works · 7,615 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 109 works

Cite

Cite as Yubo Chen, Shulin Liu, Xiang Zhang, Kang Liu and Jun Zhao. 2017. Automatically Labeled Data Generation for Large Scale Event Extraction. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1038

@inproceedings{chen2017automatically,
  title = {{Automatically Labeled Data Generation for Large Scale Event Extraction}},
  author = {Yubo Chen, Shulin Liu, Xiang Zhang, Kang Liu and Jun Zhao},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1038},
  url = {https://doi.org/10.18653/v1/p17-1038}
}

References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Reasoning About Time and Events. pages 1–8. http://dl.acm.org/citation.cfm?id=1629235.1629236. ↗
  2. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics. Association for Computational Linguistics, pages 86–90.…
  3. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1533–1544. http://aclweb.org/antholog… ↗
  4. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data. pages 1247–1250. http://doi.acm.org/10.1145/137… ↗
  5. Nathanael Chambers and Dan Jurafsky. 2011. Template-based information extraction without the templates. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 976–986. http://aclweb.org… ↗
  6. Conclusion and Future Work Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of the 53rd Annual Meet-In this paper, we present an approach to automatically label training data for EE. The experim…
  7. Computational Natural Language Learning. Association for Computational Linguistics, pages 754–765. http://aclweb.org/anthology/D12-1069. ↗
  8. Qi Li, Heng Ji, Yu Hong, and Sujian Li. 2014. Constructing information networks using one single model. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1846–1851. https://doi.org/10.3115/v1/D14-1198. ↗
  9. Kit Jackie Chi Cheung, Hoifung Poon, and Lucy Vanderwende. 2013. Probabilistic frame induction. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 837–846.… find
  10. Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 73– 82. http://aclweb.org/anthology/P13-1008. ↗
  11. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 https://arxiv.org/pdf/1207.0580. ↗
  12. Shulin Liu, Yubo Chen, Shizhu He, Kang Liu, and Jun Zhao. 2016. Leveraging framenet to improve automatic event detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 2134–2143. http://www.aclweb.org/anth… ↗
  13. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and S. Daniel Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Associ… find
  14. David McClosky, Mihai Surdeanu, and Christopher Manning. 2011. Event extraction as dependency parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 1626–1635. http://aclweb.or… ↗
  15. Yu Hong, Jianfeng Zhang, Bin Ma, Jianmin Yao, Guodong Zhou, and Qiaoming Zhu. 2011. Using cross-entity inference to improve event extraction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguis…
  16. Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. Assoc… find
  17. Lifu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji, Clare R. Voss, Jiawei Han, and Avirup Sil. 2016. Liberal event extraction and event schema induction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 258–… find
  18. Huu Thien Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Association for C…
  19. Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of ACL-08: HLT. Association for Computational Linguistics, pages 254–262. http://aclweb.org/anthology/P08-1030. ↗
  20. Sebastian Krause, Hong Li, Hans Uszkoreit, and Feiyu Xu. 2012. Large-scale learning of relation-extraction rules with distant supervision from the web. In Proceedings of International Semantic Web Conference, Springer, pages 263– 278. http://link.springer.com/chapter/10.1007/978-3-642-35176-1 17. — [doi:10.1007/978-3-642-35176-1](https://doi.org/10.1007/978-3-642-35176-1) ↗
  21. Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistic…
  22. Jayant Krishnamurthy and Tom Mitchell. 2012. Weakly supervised training of semantic parsers. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Kevin Reschke, Martin Jankowiak, Mihai Surdeanu, Christopher D Manning, and Daniel Jurafsky. 2014. Event ex… find
  23. Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and D. Christopher Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association… find
  24. Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 384–394. http://aclwe…
  25. Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016. Relation classification via multi-level attention cnns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 1298–1307. http://www.aclweb.org/anthology… ↗
  26. Matthew D Zeiler. 2012. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701 https://arxiv.org/pdf/1212.5701. ↗
  27. Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 1753–1762. https… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2017-automatically-labeled-data-generation-large/