Elamur Research⌕
☰
home / papers / 2026 / A Qualia-Based Audit of Procedural Event Annotations

A Qualia-Based Audit of Procedural Event Annotations

Paper · 2026 · Kyeongmin Rim, Marc Verhagen and James Pustejovsky

Summary

Crowdsourced open-vocabulary annotations of entity state changes, such as those in OpenPI, mix changes that matter for a procedure with incidental ones. In a recipe, an oven getting hot is recorded alongside a cake becoming baked. The authors propose a symbolic audit based on Generative Lexicon theory. They build Entity Qualia Structure (EQS), a per-entity record with two features:

  • a coarse sortal type (natural, artifactual, or instrument) derived from the FORMAL quale;
  • a flag for whether the AGENTIVE quale lists a specific creation activity.

Sortal types come from a cascade over BSO, CoreLex, and WordNet, where the first resource that gives an answer wins. On the OpenPI food domain, about half of the transformation annotations concern food itself. Nearly a third track instruments, which can be detected from entity type alone. EQS flags a share of annotations that neither human re-annotation nor LLM-based filtering identifies. The agentive feature turns out to be uninformative unless it is combined with verb-side event semantics. The work offers an ontology-grounded quality check that complements judgment-based cleanup, and it comes with a public release of BSO.

Key points

  • EQS covers 84.7% of the 518-item OpenPI food entity vocabulary, assigning 205 natural, 118 artifactual, and 116 instrument types. The three resources disagree on 32.2% of items, and the authors use this rate as a built-in quality signal.
  • Of 9367 food transformation annotations, 51.1% concern food entities and 30.2% record instrument state changes.
  • In a three-way comparison, EQS is the only method to flag 15.6% of annotations; OpenPI-C (human re-annotation) and OpenPI 2.0 (LLM salience scoring) both miss them.
  • On a stratified sample, FORMAL precision is 0.918 for instruments and 0.625 for food. Estimated recall is 0.575, giving F1 0.707.
  • Among annotations whose entity is agentive-positive, 93% involve instruments such as knives. Detecting creation therefore requires pairing the AGENTIVE quale with the verb's event semantics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brandeis UniversityWaltham · United States32 works · 5,506 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Kyeongmin Rim, Marc Verhagen and James Pustejovsky. 2026. A Qualia-Based Audit of Procedural Event Annotations. Proceedings of the 9th Workshop on Event Extraction and Understanding: Challenges and Applications (EEUCA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eeuca-1.6

@inproceedings{rim2026qualia,
  title = {{A Qualia-Based Audit of Procedural Event Annotations}},
  author = {Kyeongmin Rim, Marc Verhagen and James Pustejovsky},
  booktitle = {Proceedings of the 9th Workshop on Event Extraction and Understanding: Challenges and Applications (EEUCA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.eeuca-1.6},
  url = {https://doi.org/10.18653/v1/2026.eeuca-1.6}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nuria Bel, Federica Busa, Nicoletta Calzolari, Elisabetta Gola, Alessandro Lenci, Monica Monachini, Antoine Ogonowski, Ivonne Peters, Wim Peters, Nilda Ruimy, Marta Villegas, and Antonio Zampolli. 2000. SIM-PLE: A general framework for the development of multilingual lexicons. In Proceedings of the… find
  2. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71– 106.
  3. Antonio Toral and Monica Monachini. 2007. SIMPLE-OWL: A Generative Lexicon ontology for NLP and the semantic web. In Proceedings of the workshop on GL2007. find
  4. James Pustejovsky. 1995. The Generative Lexicon. MIT Press, Cambridge, MA. find
  5. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  6. James Pustejovsky. 2001. Type construction and the logic of concepts. In Pierrette Bouillon and Federica Busa, editors, The Language of Word Meaning, pages 91–123. Cambridge University Press. find
  7. Xueqing Wu, Sha Li, and Heng Ji. 2023. OpenPI-C: A better benchmark and stronger baseline for open-vocabulary state tracking. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7213–7222, Toronto, Canada. Association for Computational Linguistics. find
  8. James Pustejovsky, Catherine Havasi, Jessica Littman, Anna Rumshisky, and Marc Verhagen. 2006. Towards a generative lexical resource: The Brandeis semantic ontology. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language R… find
  9. Ichiro Yamada and Timothy Baldwin. 2004. Automatic discovery of Telic and Agentive roles from corpus data. In Proceedings of the 18th Pacific Asia Conference on Language, Information and Computation (PACLIC). find
  10. James Pustejovsky, Jessica Moszkowicz, Olga Batiukova, and Anna Rumshisky. 2009. GLML: Annotating argument selection and coercion. In Proceedings of the Eight International Conference on Computational Semantics, pages 169–180, Tilburg, The Netherlands. Association for Computational Linguistics. find
  11. Bingyang Ye, Jingxuan Tu, and James Pustejovsky. 2025. Enhanced noun-noun compound interpretation through textual enrichment. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25896–25911, Suzhou, China. Association for Computational Linguistics. find
  12. Li Zhang, Hainiu Xu, Abhinav Kommula, Chris Callison-Burch, and Niket Tandon. 2024. OpenPI2.0: An improved dataset for entity tracking in texts. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 166–178, St.… find
  13. James Pustejovsky, Anna Rumshisky, Alex Plotnick, Elisabetta Jezek, Olga Batiukova, and Valeria Quochi. 2010. SemEval-2010 task 7: Argument selection and coercion. In Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval), pages 27–32. find
  14. Kyeongmin Rim and James Pustejovsky. 2026. Subevent structure as a predictor of entity identity change in procedural text. In Proceedings of the Workshop on Structured Linguistic Data and Evaluation (SLiDE).
  15. Kyeongmin Rim, Jingxuan Tu, Bingyang Ye, Marc Verhagen, Eben Holderness, and James Pustejovsky. 2023. The coreference under transformation labeling dataset: Entity tracking in procedural texts using event models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12448– 12… find
  16. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, pages 4444–4451. find
  17. Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. 2020. A dataset for tracking entities in open domain procedural text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rim-2026-qualia-based-audit-procedural-event/