Elamur Research⌕
☰
home / papers / 2026 / Vrittanta-EN: A Benchmark Dataset for Event Trigger Detection and Classification Advancing Event Understanding in English Narrative Discourse

Vrittanta-EN: A Benchmark Dataset for Event Trigger Detection and Classification Advancing Event Understanding in English Narrative Discourse

Paper · 2026 · Chaitanya Kirti, Ashish Anand and Prithwijit Guha

Summary

Most event extraction resources for English cover factual text such as news or biomedical reports. Stories have received much less attention, even though their events often surface through dialogue, feelings and perception rather than explicit factual markers. Vrittanta-EN addresses this with a manually annotated English corpus for two subtasks: finding the words that signal an event (trigger detection) and assigning each trigger to a type (classification). The corpus is built from 200 short stories taken from Indian fable and moral-tale collections. It contains 11,272 event instances, each labeled with one of seven narrative-oriented classes. Only events that actually happen are annotated.

The authors benchmark a broad set of systems on the corpus:

  • classical classifiers
  • recurrent taggers
  • encoder transformers
  • encoder-decoder models
  • prompted LLMs in zero-shot and five-shot setups

Fine-tuned contextual encoders come out on top. The best detector is ELECTRA and the best classifier is RoBERTa, while class-level scores lag clearly behind detection. The resource gives narrative event understanding a dedicated testbed. It also exposes recurring weak spots: multiword triggers, hypothetical events and rare classes.

Key points

  • Corpus and annotation: 11,272 annotated event instances from 200 stories. The stories are drawn from eight collections: the Jataka Tales and Singhasan Battisi, the Hitopadesh and Panchatantra, Betal Pachisi, the magazine Champak, and the tales of Akbar–Birbal and Tenali Raman. Annotation was done in BRAT by two trained annotators.
  • Classes: seven types.
    • COMMUNICATION
    • GENERAL-ACTIVITY
    • MOVEMENT
    • COGNITIVE-MENTAL-STATE
    • LIFE-EVENT
    • CONFLICT
    • OTHERS (a residual catch-all)
  • What counts as a trigger: finite and non-finite verbs, participles, nominalizations (e.g. "marriage") and emotional or resultant adjectives (e.g. "terrified"). Non-occurring actions and sub-events embedded inside a larger event are excluded.
  • Models tested:
    • classical: SVM, Naive Bayes
    • recurrent: LSTM, BiLSTM, BiLSTM-CRF
    • encoders: BERT, RoBERTa, ALBERT, DistilBERT, DeBERTa, ELECTRA
    • encoder-decoders: T5, BART
    • LLMs: GPT-4.1, DeepSeek-V3.2-Exp, Claude Sonnet 4, each in zero-shot and five-shot settings
  • Best results: ELECTRA reaches 90.61% F1 for detection. RoBERTa reaches 74.71% macro F1 for classification.
  • Typical errors:
    • detection: phrasal triggers cut short ("made up" predicted as "made"), hypothetical verbs under modals or "if" marked as events, and implicit state events like "rejoiced" missed
    • classification: confusions among GENERAL-ACTIVITY, MOVEMENT and OTHERS, cognition mistaken for communication, and rare classes (LIFE-EVENT, CONFLICT) absorbed into frequent ones

Interesting ideas

  • I1. The corpus annotates only realis events, so an action under a conditional or modal does not count as having happened; this cleanly separates "what occurred" from "what was imagined."
  • I2. Sub-events nested inside a broader event are deliberately left unmarked, which keeps the event list from being inflated by components of a single occurrence.
  • I3. The most frequent detection failure is truncated multiword triggers such as phrasal verbs, where the meaning depends on the whole phrase.
  • I4. The error analysis groups failures into named types (boundary, hypothetical false positive, implicit trigger, class overlap) instead of reporting a single score.
  • I5. The authors propose a hierarchical class organization and semantic-role cues as ways to reduce confusion between neighboring classes like motion versus general activity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Only realis frames should change entity states and the event order. Frames governed by an if mark or a could/would aux in the UD tree should go into a separate hypothetical layer.
  • I2 (frames): A comparable rule could fold embedded or complement verbs into their governing frame, instead of emitting parallel "who did what" frames.
  • I3 (tree): Treating compound:prt and similar UD relations as part of the predicate head before frame lookup would avoid mapping "made up (one's mind)" to plain "make."
  • I4 (learn): We could build targeted negative-control sets for each type, such as conditional sentences that must yield no realized event, and track them separately during evaluation.
  • I5 (frames): Our VerbNet-based roles and classes could give a principled path from fine verb classes to coarse narrative types, which helps fable QA tell thinking apart from telling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 worksDiscourse analysis · 466 works

Cite

Cite as Chaitanya Kirti, Ashish Anand and Prithwijit Guha. 2026. Vrittanta-EN: A Benchmark Dataset for Event Trigger Detection and Classification Advancing Event Understanding in English Narrative Discourse. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/25s8ijb2orwm

@inproceedings{kirti2026vrittanta,
  title = {{Vrittanta-EN: A Benchmark Dataset for Event Trigger Detection and Classification Advancing Event Understanding in English Narrative Discourse}},
  author = {Chaitanya Kirti, Ashish Anand and Prithwijit Guha},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/25s8ijb2orwm},
  url = {https://doi.org/10.63317/25s8ijb2orwm}
}

References

35 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacqueline Aguilar, Charley Beller, Paul McNamee, Stephanie Strassel, and Zhiyi Song. 2014. A comparison of the events and relations across ace, ere, and tac-kbp. In Workshop on Events: Definition, Detection, Coreference, and Representation. find
  2. David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Reasoning about Time and Events.
  3. David Bamman, Ted Underwood, et al. 2020. An annotated dataset of literary entities. In Proceedings of LREC.
  4. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of EMNLP. ↗
  5. Tommaso Caselli et al. 2017. Event storyline corpus: A new benchmark for causal and temporal relation extraction. In Proceedings of ACL. find
  6. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pages 789– 797. find
  7. Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. 2024. Is a large language model a good annotator for event extraction? In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 17772– 17780. find
  8. Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of ACL.
  9. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. ↗
  11. Tommaso D’Odorico. 2013. An ontological analysis of vague motion verbs, with an application to event recognition. University of Leeds. find
  12. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654. ↗
  13. Pengcheng He et al. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In ICLR. ↗
  14. Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In Proceedings of ACL.
  15. Jin-Dong Kim, Ngan Nguyen, Yue Wang, Jun’ichi Tsujii, Toshihisa Takagi, and Akinori Yonezawa. 2012. The genia event and protein coreference tasks of the bionlp shared task 2011. BMC bioinformatics, 13(Suppl 11):S1. ↗
  16. Magnus Knuth, Jens Lehmann, Dimitris Kontokostas, Thomas Steiner, and Harald Sack. 2015. The dbpedia events dataset. In ISWC (Posters & Demos). find
  17. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  18. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: A pre-trained biomedical language representation model for biomedical text mining. In Bioinformatics. ↗
  19. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
  20. Mike Lewis et al. 2020b. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL.
  21. Sha Li et al. 2021. Future is not one-dimensional: Graph modeling of event structures for event prediction. In Proceedings of ACL. find
  22. Xiaodong Liu, Zhunchen Luo, and Heyan Huang. 2018. Jointly multiple events extraction via attention-based graph information aggregation. In Proceedings of EMNLP.
  23. Yinhan Liu et al. 2019. Roberta: A robustly optimized bert pretraining approach. In arXiv preprint arXiv:1907.11692. ↗
  24. Teruko Mitamura and Zhengzhong Liu. Overview of tac kbp 2015 event nugget track. find
  25. Teruko Mitamura et al. 2016. Overview of tac kbp 2016 event track. In Text Analysis Conference (TAC). find
  26. Thien Huu Nguyen and Ralph Grishman. 2015. Event detection and domain adaptation with convolutional neural networks. In Proceedings of ACL.
  27. James Pustejovsky and Amber Stubbs. 2019. Towards a standard for annotating events in text. In Proceedings of LREC. find
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020a. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. ↗
  29. Colin Raffel, Noam Shazeer, et al. 2020b. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR. ↗
  30. Priyanka Ranade, Sanorita Dey, Anupam Joshi, and Tim Finin. 2022. Computational understanding of narratives: A survey. IEEE Access, 10:101575–101594. ↗
  31. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. ↗
  32. Matthew Sims, Jong Ho Park, and David Bamman. 2019. Literary event detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3623–3634. find
  33. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. brat: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations Session at EACL 2012, Avignon, France. Association for Computational Linguistics.
  34. Ozlem Uzuner, Peter Szolovits, and Isaac Kohane. 2006. i2b2 workshop on natural language processing challenges for clinical records. In Proceedings of the Fall Symposium of the American Medical Informatics Association. Citeseer. find
  35. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2005. The ace 2005 (automatic content extraction) evaluation. In Proceedings of the Linguistic Data Consortium (LDC). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kirti-2026-vrittanta-en-benchmark-dataset-event/