Elamur Research⌕
☰
home / papers / 2021 / Integrating Deep Event-Level and Script-Level Information for Script Event Prediction

Integrating Deep Event-Level and Script-Level Information for Script Event Prediction

Summary

The paper addresses script event prediction. Given a chain of events that have already occurred, the model must choose the next event from several candidates. The authors point out two shortcomings of earlier approaches. First, an event was usually reduced to a verb and the head words of its participants, so other useful contextual details were lost. Second, only a single event sequence centered on one protagonist was considered. To address both shortcomings, they propose MCPredictor, a Transformer-based model. At the level of an individual event, it combines a vector representation of the tuple (verb and participants) with an encoding of the source sentence produced by BERT-tiny. At the script level, the model builds a separate chain for each participant of the candidate event. An attention mechanism weights the scores of these chains. Experiments were conducted on the New York Times corpus, and according to the authors, the model outperforms previous approaches. The work shows that sentence context and the perspectives of multiple participants noticeably improve event modeling.

Key points

  • An event is represented as a tuple: verb, subject, object, indirect object, and the sentence from which it was extracted. Missing participants are filled with null. Candidates have no sentence of their own.
  • The model has four components: event encoding, text encoding (the "[CLS]" output of BERT-tiny), chain modeling with a stack of Transformer layers, and attention-based scoring that aggregates multiple chains. To prevent information leakage, the other events of the chain in the sentence are replaced with "[UNK]", and the protagonist's role is marked with the tags [subj]/[obj]/[iobj].
  • On the NYT development set, the full model with all sentence components achieves an accuracy of 67.35 %. Masking verbs lowers accuracy by 6.81 points, masking nouns by 2.54, and masking both verbs and nouns by 11.99.
  • Masking the verb of the event itself has almost no effect on the result (−0.06), because this information is already present in the tuple encoding. In contrast, masking the remaining verbs in the sentence reduces accuracy by 6.45 points.
  • Among the limitations, the authors note that the candidate's participants must be known in advance. Otherwise, combinations of entities have to be enumerated, which is costly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.777 · arXiv 2110.15706

@inproceedings{bai2021integrating,
  title = {{Integrating Deep Event-Level and Script-Level Information for Script Event Prediction}},
  author = {Bai, Long and Guan, Saiping and Guo, Jiafeng and Li, Zixuan and Jin, Xiaolong and Cheng, Xueqi},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.777},
  eprint = {2110.15706},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.777}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Robert Abelson and Roger C Schank. 1977. Scripts, plans, goals and understanding. An inquiry into human knowledge structures New Jersey, 10. find
  2. Niranjan Balasubramanian, Stephen Soderland, Mausam, and Oren Etzioni. 2013. Generating coherent event schemas at scale. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1721–1731, Seattle, Washington, USA. Association for Computational Linguistics. find our card
  3. David Bean and Ellen Riloff. 2004. Unsupervised learning of contextual role knowledge for coreference resolution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 297–304, Boston, Massachu… find our card
  4. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789–797, Columbus, Ohio. Association for Computational Linguistics. find
  5. Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 602–610, Suntec, Si… find
  6. Snigdha Chaturvedi, Haoruo Peng, and Dan Roth. 2017. Story comprehension for predicting what happens next. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1603–1614, Copenhagen, Denmark. Association for Computational Linguistics. find
  7. James Curran, Stephen Clark, and Johan Bos. 2007. Linguistically motivated large-scale NLP with c&c and boxer. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 33–36, Prague, Czech Republic.… find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  9. Xiao Ding, Kuo Liao, Ting Liu, Zhongyang Li, and Junwen Duan. 2019. Event representation learning enhanced with external commonsense knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro… find
  10. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34. find
  11. Mark Granroth-Wilding and Stephen Clark. 2016. What happens next? event prediction using a compositional neural network model. find
  12. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, I-CLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find
  14. I-Ta Lee and Dan Goldwasser. 2019. Multi-relational script learning for discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4214–4226, Florence, Italy. Association for Computational Linguistics. find our card
  15. Zhongqing Wang, Yue Zhang, and Ching-Yun Chang. 2017. Integrating order information and event relation for script event prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 57–67, Copenhagen, Denmark. Association for Computational Linguistics. find
  16. I-Ta Lee, Maria Leonor Pacheco, and Dan Goldwasser. 2020. Weakly-supervised modeling of contextualized event embedding for discourse relations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4962–4972, Online. Association for Computational Linguistics. find our card
  17. Jianming Zheng, Fei Cai, and Honghui Chen. 2020. Incorporating scenario knowledge into a unified fine-tuning architecture for event representation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 249–258. find
  18. Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Constructing narrative event evolutionary graph for script event prediction. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4201–4207. International Joint Conferences on Artificial Intellige… find
  19. Shangwen Lv, Wanhui Qian, Longtao Huang, Jizhong Han, and Songlin Hu. 2019. Sam-net: Integrating event-level and chain-level attentions to predict what happens next. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Inte… find
  20. Shangwen Lv, Fuqing Zhu, and Songlin Hu. 2020. Integrating external event knowledge for script learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 306–315, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  21. Ashutosh Modi and Ivan Titov. 2014. Learning semantic script knowledge with event embeddings. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings. find
  22. Karl Pichotta and Raymond Mooney. 2014. Statistical script learning with multi-argument events. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 220–229, Gothenburg, Sweden. Association for Computational Linguistics. find
  23. Karl Pichotta and Raymond Mooney. 2016. Learning statistical scripts with lstm recurrent neural networks. find

↑ back to top

This page in Markdown →