Elamur Research⌕
☰
home / papers / 2026 / Missing Links: LLM-Augmentation of Event Triggers of State Changes in the OpenPI Dataset

Missing Links: LLM-Augmentation of Event Triggers of State Changes in the OpenPI Dataset

Summary

Most large procedural datasets, such as OpenPI, record what happens to entities (state-change tuples of entity, attribute, before-state and after-state) but not which words express the causing action. Without those event triggers, the datasets cannot support joint modeling of actions and their effects, as in the POEM/CuT framework. The authors propose "inverse annotation," which runs the usual annotation direction in reverse. An LLM receives OpenPI's existing state-change tuples and proposes the trigger words likely to have produced them. The pipeline has four stages. Documents without real multi-step tasks are filtered out first. An LLM then generates five trigger candidates with example sentences. Candidates are scored with GloVe and Sentence-BERT similarity against the original step. Finally, surviving triggers are aligned to character spans in the source sentence. The result is OpenPI-A, a silver-standard trigger layer, which the authors benchmark with statistical, rule-based, frame-semantic and neural detectors. They also report that a model's generation success rate does not predict annotation quality. The work offers both a reusable recovery method and a new resource linking actions to their outcomes.

Key points

  • The source data is OpenPI: 810 WikiHow documents, 4,050 steps and 31,473 state-change annotations across six domains. Only the train split (644 documents, 3,216 steps) was used.
  • The final OpenPI-A set has 515 examples, built as the union of outputs from gemma-3n-4b (382) and qwen3-14b (343) at threshold 0.80. On the 191 sentences both models annotated, they agreed on 172 (90.0%).
  • Span alignment was mostly exact match (88.9%), with lemma match at 5.6% and semantic match at 5.4%. The most frequent triggers are "cut" (25) and "add" (22).
  • Baselines include EVITA, a VerbNet parser, few-shot LLMs and RoBERTa-base fine-tuned on MAVEN. The RoBERTa model reaches 95.91% token-level F1 on MAVEN validation, a score not comparable to the zero-shot evaluation on OpenPI-A.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4ga5mnybeeam

@inproceedings{rim2026missing,
  title = {{Missing Links: LLM-Augmentation of Event Triggers of State Changes in the OpenPI Dataset}},
  author = {Rim, Kyeongmin and Pustejovsky, James},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4ga5mnybeeam},
  url = {https://doi.org/10.63317/4ga5mnybeeam}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet Project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1, ACL ’98, pages 86–90, Montreal, Quebec, C… find our card
  2. Susan Windisch Brown, Julia Bonn, James Gung, Annie Zaenen, James Pustejovsky, and Martha Palmer. 2019. VerbNet Representations: Subevent Semantics for Transfer Verbs. In Proceedings of the First International Workshop on Designing Meaning Representations, pages 154– 163, Florence, Italy. Associatio… find our card
  3. Young Min Cho, Li Zhang, and Chris Callison-Burch. 2022. Unsupervised Entity Linking with Guided Summarization and Multiple-Choice Selection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9394–9401, Abu Dhabi, United Arab Emirates. Association for C… find
  4. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… find
  5. Kenneth Ward Church and Patrick Hanks. 1990. Word Association Norms, Mutual Information, and Lexicography. Computational Linguistics, 16(1):22–29. find our card
  6. Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis… find
  7. Mark Davies. 2008. The Corpus of Contemporary American English (COCA). find
  8. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbo… find our card
  9. Weihong Du, Wenrui Liao, Hongru Liang, and Wenqiang Lei. 2024. PAGED: A Benchmark for Procedural Graphs Extraction from Documents. find
  10. James Gung and Martha Palmer. 2021. Predicate Representations and Polysemy in VerbNet Semantic Parsing. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 51–62, Groningen, The Netherlands (online). Association for Computational Linguistics. find
  11. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, Adriane Boyd, and Henning Peters. 2023. spaCy: Industrial-strength Natural Language Processing in Python. Zenodo. find
  12. Xiaomeng Jin and Heng Ji. 2024. Schema-based Data Augmentation for Event Extraction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14382–14392, Torino, Italia. ELRA and ICCL. find
  13. Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. 2023. Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction. find
  14. Ghazaleh Kazeminejad, Martha Palmer, Tao Li, and Vivek Srikumar. 2021. Automatic Entity State Annotation using the VerbNet Semantic Parser. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 123–132, Punta Cana, Domin… find
  15. Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008. A large-scale classification of English verbs. Language Resources and Evaluation, 42(1):21–40. find
  16. Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian, and Raymond Mooney. 2025. CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans. find
  17. Mingyu Derek Ma, Xiaoxuan Wang, Po-Nien Kung, P. Jeffrey Brantingham, Nanyun Peng, and Wei Wang. 2024. STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18751–18759… find
  18. Abhilash Nandy, Yash Kulkarni, Pawan Goyal, and Niloy Ganguly. 2024. Order-Based Pre-training Strategies for Procedural Text Understanding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Sho… find
  19. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  20. James Pustejovsky, Robert Knippen, Jessica Littman, and Roser Saurí. 2005. Temporal and Event Information in Natural Language Text. Language Resources and Evaluation, 39(2):123– 164. find
  21. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
  22. Kyeongmin Rim, Jingxuan Tu, Bingyang Ye, Marc Verhagen, Eben Holderness, and James Pustejovsky. 2023. The Coreference under Transformation Labeling Dataset: Entity Tracking in Procedural Texts Using Event Models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12448–124… find
  23. Roser Saurí, Robert Knippen, Marc Verhagen, and James Pustejovsky. 2005. Evita: A Robust Event Recognizer For QA Systems. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 700–707, Vancouver, British Columbia, Canada. Ass… find
  24. Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. 2020. A Dataset for Tracking Entities in Open Domain Procedural Text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),… find
  25. Mashiro Toyooka, Kiyoharu Aizawa, and Yoko Yamakata. 2025. A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Task. find
  26. Jingxuan Tu, Timothy Obiso, Bingyang Ye, Kyeongmin Rim, Keer Xu, Liulu Yue, Susan Windisch Brown, Martha Palmer, and James Pustejovsky. 2024. GLAMR: Augmenting AMR with GL-VerbNet Event Structure. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resour… find
  27. Marc Verhagen and James Pustejovsky. 2012. The TARSQI Toolkit. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2043–2048, Istanbul, Turkey. European Language Resources Association (ELRA). find
  28. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1652–1671, Onlin… find our card
  29. Te-Lin Wu, Caiqi Zhang, Qingyuan Hu, Alexander Spangher, and Nanyun Peng. 2023. Learning Action Conditions from Instructional Manuals for Instruction Understanding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3023–3043, To… find
  30. Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. 2024. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 18(6):186357. find
  31. Yoko Yamakata, Shinsuke Mori, and John Carroll. 2020. English Recipe Flow Graph Corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5187–5194, Marseille, France. European Language Resources Association. find
  32. Li Zhang, Hainiu Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora, and Chris Callison-Burch. 2023. Causal Reasoning of Entities and Events in Procedural Texts. In Findings of the Association for Computational Linguistics: EACL 2023, pages 415–431, Dubrovnik, Croatia. Association for Computational… find
  33. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A Robustly Optimized BERT Pre-training Approach with Post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227, Huhhot, China. Chinese Information Processing Society of China. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rim-2026-missing-links-llm-augmentation-event/