Elamur Research⌕
☰
home / papers / 2024 / MAVEN-FACT: A Large-scale Event Factuality Detection Dataset

MAVEN-FACT: A Large-scale Event Factuality Detection Dataset

Paper · 2024 · Chunyang Li, Hao Peng, Xiaozhi Wang et al.

Summary

Event Factuality Detection (EFD) asks whether a mentioned event actually happened, might happen, or did not happen. Progress has been held back by small datasets that lack other event annotations. FactBank, the most common benchmark, covers only 9,761 events. The authors build MAVEN-FACT on top of the MAVEN documents and label 112,276 events, which makes it the largest EFD resource. It uses five factuality classes (CT+, PS+, PS-, CT-, Uu) and marks the words that signal non-factuality.

To keep costs down, they use an LLM-then-human pipeline. GPT-3.5 screens out clearly factual training events, and humans label everything else. Validation and test events are fully human-annotated. Because MAVEN already carries event types, arguments and relations, the same events can be studied jointly with these layers.

The benchmark is hard. The best system reaches only 47.6% macro F1, and models are even weaker at pointing to the evidence words. Argument and relation information helps fine-tuned models but not LLMs. Supplying factuality to LLMs reduces event-related hallucination. This matters for any system that must avoid treating hypothetical events as real.

Key points

  • MAVEN-FACT has factuality labels for 112,276 events, inherited from MAVEN together with MAVEN's event types, arguments (MAVEN-Arg) and relations (MAVEN-ERE).
  • The label scheme is the FactBank-style 5-way set, combining modality (CT certain vs. PS possible) with polarity (+/-), plus Uu for undeterminable cases. Evidence words are annotated only for PS+, PS- and CT- events.
  • GPT-3.5 pre-annotation was framed as binary CT+ vs. non-CT+ with a rule-based chain-of-thought prompt.
    • In a 500-event pilot, it caught about 97.5% of non-CT+ events while still clearing 16.5% of CT+ events.
    • On the training set, 16,950 events were auto-labelled CT+ and 56,989 went to human annotators (47 in total).
    • This saved roughly 15% of the cost, about 2,500 USD, and a 50-document audit found under 2% noise.
  • Models evaluated include fine-tuned RoBERTa-style EFD models and in-context Mistral 7B, LLAMA 3, GPT-3.5 and GPT-4. Even the best reaches only 47.6% macro F1, and evidence-word F1 is much lower.
  • In a hallucination study on 450 validated questions, adding gold factuality improved accuracy for LLAMA 3 and GPT-4. Factuality predicted by DMRoBERTa also helped. The test set stays private behind an online scorer to limit contamination.

Interesting ideas

  • I1. Use a cheap automatic filter tuned for very high recall on the rare, hard class, so that only easy majority cases are auto-labelled and everything uncertain goes to humans; this works wherever one label dominates.
  • I2. Write the annotation prompt as an explicit step-by-step procedure: identify whose perspective the event is reported from, look for modal words, check for negation, and default to "uncertain" when unsure; a safe default plus an ordered checklist makes the decisions auditable.
  • I3. Annotate the trigger words that justify a non-factual label, and score them separately, because a model can pick the right class for the wrong reason.
  • I4. Encode factuality as two orthogonal axes, certainty and polarity, plus an explicit "cannot tell" value, instead of one fuzzy score; this gives a small label set that supports reasoning downstream.
  • I5. Measure the downstream benefit with both gold and predicted factuality: the gold setting gives an upper bound and the predicted setting shows real-world gain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver-data agreement could auto-accept only the overwhelmingly common, low-risk decisions, such as plain asserted past events, and send every hedged or contested case to the LLM teacher tier.
  • I2 (frames): The same ordered checklist can be a deterministic rule pass over UD trees, finding modal aux, neg, conditional mark and attitude verbs, that assigns each frame a CT/PS ± status with a logged reason.
  • I3 (answer): Answers that depend on whether an event happened should cite the cue token ("might", "provided that"), and evaluation should check that the cited cue is correct, not just the final yes/no.
  • I4 (world): World-of-text event records could carry a (certainty, polarity, source) triple, so that possible or negated events never become asserted states or holdings.
  • I5 (learn): Mova's evaluation could run each answering test twice, with gold and with predicted factuality, to see how much reader error comes from the factuality step rather than from later reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu and Juanzi Li. 2024. MAVEN-FACT: A Large-scale Event Factuality Detection Dataset. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-emnlp.651

@inproceedings{li2024maven,
  title = {{MAVEN-FACT: A Large-scale Event Factuality Detection Dataset}},
  author = {Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu and Juanzi Li},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.651},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.651}
}

References

60 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. 2024. Many-shot in-context learning. arXiv preprint arXiv:2404.11018. ↗
  2. Adnan Akhundov, Dietrich Trautmann, and Georg Groh. 2018. Sequence labeling: A practical approach. arXiv preprint arXiv:1808.03926. ↗
  3. David Alvarez Melis and Tommi Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Proceedings of NeurIPs. ↗
  4. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. In Proceedings of NeurIPs. 0/ https://creativecommons.org/licenses/by-sa/4. https://www.gnu.org/licenses/gpl-3.0.html ↗
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of NeurIPs, pages 1877– 1901.
  6. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023b. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232. ↗
  7. Pengfei Cao, Yubo Chen, Yuqing Yang, Kang Liu, and Jun Zhao. 2021. Uncertain local-to-global networks for document-level event factuality identification. In Proceedings of EMNLP, pages 2636–2645. find
  8. Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating hallucination in large language models via self-reflection. arXiv preprint arXiv:2310.06271. ↗
  9. Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. 2024. Is a large language model a good annotator for event extraction? In Proceedings of AAAI, pages 17772–17780. find
  10. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. ↗
  11. Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of ACL, pages 167–176.
  12. Milind Choudhary and Xinya Du. 2024. Qaevent: Event extraction as question-answer pairs generation. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1860–1873. find
  13. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. find
  14. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495. ↗
  15. Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2015. Deep learning for event-driven stock prediction. In Proceedings of IJCAI. find
  16. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234. ↗
  17. Seraphina Goldfarb-Tarrant, Haining Feng, and Nanyun Peng. 2019. Plan, write, and revise: an interactive system for open-domain story generation. In Proceedings of NAACL: Demonstrations, pages 89–97. find
  18. Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. 2023. Is information extraction solved by ChatGPT? An analysis of performance, evaluation criteria, robustness and errors. arXiv preprint arXiv:2305.14450. ↗
  19. Kuan-Hao Huang, I Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Premkumar Natarajan, Kai-Wei Chang, Nanyun Peng, Heng Ji, et al. 2023a. A reevaluation of event extraction: Past, present, and future challenges. arXiv preprint arXiv:2311.09562. ↗
  20. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  21. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of EMNLP, pages 1643–1648. find
  22. Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating ChatGPT’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304.11633. ↗
  23. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. In Proceedings of NeurIPs.
  24. Xiao Liu, Heyan Huang, and Yue Zhang. 2022. End-to-end event factuality prediction using directional labeled graph recurrent network. Information Processing & Management, 59(2):102836. ↗
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  26. Haoyan Luo and Lucia Specia. 2024. From understanding to utilization: A survey on explainability for large language models. arXiv preprint arXiv:2401.12874. ↗
  27. Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. find
  28. Anne-Lyse Minard, Manuela Speranza, Ruben Urizar, Begona Altuna, Marieke Van Erp, Anneleen Schoen, and Chantal Van Son. 2016. Meantime, the newsreader multilingual event and time corpus. In Proceedings of LREC, pages 4417–4422. find
  29. Nailia Mirzakhmedova, Marcel Gohsen, Chia Hao Chang, and Benno Stein. 2024. Are large language models reliable argument quality annotators? arXiv preprint arXiv:2404.09696. ↗
  30. John Murzaku, Tyler Osborne, Amittai Aviram, and Owen Rambow. 2023. Towards generative event factuality prediction. In Findings of ACL, pages 701– 715.
  31. OpenAI. 2022. Introducing ChatGPT. find
  32. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  33. Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou, et al. 2023a. When does in-context learning fall short and why? a study on specification-heavy tasks. arXiv preprint arXiv:2311.08993. ↗
  34. Hao Peng, Xiaozhi Wang, Feng Yao, Kaisheng Zeng, Lei Hou, Juanzi Li, Zhiyuan Liu, and Weixing Shen. 2023b. The devil is in the details: On the pitfalls of event extraction evaluation. In Findings of ACL, pages 9206–9227. find
  35. James Pustejovsky, Marc Verhagen, Roser Sauri, Jessica Littman, Robert Gaizauskas, Graham Katz, Inderjeet Mani, Robert Knippen, and Andrea Setzer. 2006. Timebank 1.2. ↗
  36. Yunjia Qi, Hao Peng, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. 2024. Adelie: Aligning large language models on information extraction. arXiv preprint arXiv:2405.05008. ↗
  37. Zhong Qian, Peifeng Li, Yue Zhang, Guodong Zhou, and Qiaoming Zhu. 2018a. Event factuality identification via generative adversarial networks with auxiliary classification. In Proceedings of IJCAI, pages 4293–4300. find
  38. Zhong Qian, Peifeng Li, Guodong Zhou, and Qiaoming Zhu. 2018b. Event factuality identification via hybrid neural networks. In Neural Information Processing: 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, December 13–16, 2018, Proceedings, Part V 25, pages 335–347. Springer. find
  39. Ryan Rifkin and Aldebaro Klautau. 2004. In defense of one-vs-all classification. The Journal of Machine Learning Research, 5:101–141. find
  40. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of NAACL-HLT, pages 731–744.
  41. Roser Saurí and James Pustejovsky. 2009. Factbank: a corpus annotated with event factuality. Language resources and evaluation, 43:227–268. ↗
  42. Roser Saurí and James Pustejovsky. 2012. Are you sure that this happened? assessing the factuality degree of events in text. Computational linguistics, 38(2):261– 299. ↗
  43. Chenglei Si, Dan Friedman, Nitish Joshi, Shi Feng, Danqi Chen, and He He. 2023. Measuring inductive biases of in-context learning with underspecified demonstrations. In Proceedings of ACL, pages 11289–11310. ↗
  44. Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019. Graph based neural networks for event factuality prediction using syntactic and semantic structures. In Proceedings of ACL, pages 4393–4399.
  45. Shichao Wang, Xiangrui Cai, Hongbin Wang, and Xiaojie Yuan. 2021. Incorporating circumstances into narrative event prediction. In Findings of EMNLP, pages 4840–4849. find
  46. Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, et al. 2022. Maven-ere: A unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction. In Proceedings of EMNLP, pages 926–941. ↗
  47. Xiaozhi Wang, Hao Peng, Yong Guan, Kaisheng Zeng, Jianhui Chen, Lei Hou, Xu Han, Yankai Lin, Zhiyuan Liu, Ruobing Xie, et al. 2023a. Maven-arg: Completing the puzzle of all-in-one event understanding dataset with event argument annotation. arXiv preprint arXiv:2311.09105. ↗
  48. Zhong Qian, Peifeng Li, Qiaoming Zhu, and Guodong Zhou. 2019. Document-level event factuality identification via adversarial neural network. In Proceedings of NAACL-HLT, pages 2799–2809. find
  49. Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. Maven: A massive general domain event detection dataset. In Proceedings of EMNLP, pages 1652–1671. ↗
  50. Zhong Qian, Peifeng Li, Qiaoming Zhu, and Guodong Zhou. 2022. Document-level event factuality identification via reinforced multi-granularity hierarchical attention networks. In Proceedings of IJCAI, pages 4338–4345. find
  51. Xiaozhi Wang, Ziqi Wang, Xu Han, Zhiyuan Liu, Juanzi Li, Peng Li, Maosong Sun, Jie Zhou, and Xiang Ren. 2019. HMEAE: Hierarchical modular event argument extraction. In Proceedings of EMNLP-IJCNLP, pages 5777–5783. find
  52. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383–2392. ↗
  53. Xingyao Wang, Sha Li, and Heng Ji. 2023b. Code4struct: Code generation for few-shot event structure prediction. In Proceedings of ACL, pages 3640–3663.
  54. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of NeurIPs, pages 24824–24837.
  55. Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. 2024. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244. ↗
  56. Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2023. Alignment for honesty. arXiv preprint arXiv:2312.07000. ↗
  57. Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024. R-tuning: Teaching large language models to refuse unknown questions. In Proceedings of NAACL. find
  58. Heng Zhang, Zhong Qian, Peifeng Li, and Xiaoxu Zhu. 2022. Evidence-based document-level event factuality identification. In Pacific Rim International Conference on Artificial Intelligence, pages 240–254. Springer. find
  59. Zihao Zhang, Zhong Qian, Xiaoxu Zhu, and Peifeng Li. 2023. Code: Contrastive learning method for document-level event factuality identification. In International Conference on Database Systems for Advanced Applications, pages 497–512. Springer. find
  60. Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1–38. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2024-maven-fact-large-scale-event/