Elamur Research⌕
☰
home / papers / 2023 / Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning

Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning

Paper · 2023 · Xiaoming Shi, Siqiao Xue, Kangrui Wang et al.

Summary

The paper asks whether large language models can help forecast real-world events in time-stamped sequences, such as political interactions between G20 entities, where each event carries text like a news headline. Classical event sequence models, typically built on intensity functions, cannot use world knowledge found in text. The authors propose LAMP, a three-stage pipeline. First, a pretrained event sequence model generates candidate predictions for the next event's time or type. Second, an LLM, guided by a handful of expert-written demonstrations, performs abductive reasoning: it proposes plausible cause events for each candidate. These generated causes then serve as queries to retrieve the most similar events that actually occurred in the history. Third, a learned ranking model scores each candidate by how well its retrieved evidence supports it. The framework works with any event sequence model and any LLM. On several real-world datasets, it significantly outperforms state-of-the-art event sequence models. The authors present this as the first integration of LLMs into event sequence modeling.

Key points

  • LAMP has three components: a base event sequence model that proposes candidates, an LLM that suggests causes and drives retrieval, and a neural ranking model that weighs the retrieved evidence.
  • For time prediction, the system draws L samples using the thinning algorithm and adds the MBR estimate, giving L+1 candidates. For type prediction, it keeps the top-M types by intensity. With structured types such as subject–predicate–object, only the unknown attribute (for example, the predicate) varies across candidates.
  • The few-shot prompt asks the LLM to produce four or five cause events per effect event, with predicates limited to 20 options. The LLMs referenced are GPT-3.0-davinci (training data through October 2019) and GPT-3.5-turbo (through September 2021).
  • The reasoning step is a single backward-chaining-like hop that assumes fully observed data. Handling incomplete histories, where indirect causes matter, is left for future work.
  • The work was published at NeurIPS 2023 by authors from Ant Group, UChicago and TTIC. The provided excerpt omits the dataset names and numerical results, so specific metrics are not reported here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Y. Zhang, Jun Zhou, Chenhao Tan and Hongyuan Mei. 2023. Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning. arXiv:2305.16646.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.16646 · arXiv 2305.16646

@misc{shi2023language,
  title = {{Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning}},
  author = {Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Y. Zhang, Jun Zhou, Chenhao Tan and Hongyuan Mei},
  year = {2023},
  doi = {10.48550/arxiv.2305.16646},
  eprint = {2305.16646},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.16646}
}

References

83 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ballesteros, M., Anubhai, R., Wang, S., Pourdamghani, N., Vyas, Y., Ma, J., Bhatia, P., McKeown, K., and Al-Onaizan, Y. Severing the edge between before and after: Neural architectures for temporal ordering of events. In Proceedings of the Conference on Empirical Methods in Natural Language Processi… find
  2. Boschee, E., Lautenschlager, J., O’Brien, S., Shellman, S., Starz, J., and Ward, M. ICEWS Coded Event Data, 2015. find
  3. Boyd, A., Bamler, R., Mandt, S., and Smyth, P. User-dependent neural sequence models for continuous-time event data. In Advances in Neural Information Processing Systems (NeurIPS), 2020. find
  4. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  5. Chan, Y. S. and Roth, D. Exploiting syntactico-semantic structures for relation extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2011.
  6. Chen, M., Zhang, H., Ning, Q., Li, M., Ji, H., McKeown, K., and Roth, D. Event-centric natural language processing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2021. find
  7. Choi, E., Du, N., Chen, R., Song, L., and Sun, J. Constructing disease network and temporal progression model via context-sensitive Hawkes process. In IEEE International Conference on Data Mining (ICDM), 2015. find
  8. Deng, S., Rangwala, H., and Ning, Y. Dynamic knowledge graph based multi-event forecasting. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020. find
  9. Deng, S., Rangwala, H., and Ning, Y. Understanding event predictions via contextualized multilevel feature learning. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, pp. 342–351, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978145… find
  10. Deng, S., Rangwala, H., and Ning, Y. Causality enhanced societal event forecasting with heterogeneous graph learning. In 2022 IEEE International Conference on Data Mining (ICDM), 2022. find
  11. Do, Q., Lu, W., and Roth, D. Joint inference for event timeline construction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2012. find
  12. Dror, R., Wang, H., and Roth, D. Zero-shot on-the-fly event schema induction. In Findings of the Conference of the European Chapter the Association for Computational Linguistics (Findings of EACL), 2023. find
  13. Du, N., Farajtabar, M., Ahmed, A., Smola, A. J., and Song, L. Dirichlet-Hawkes processes with applications to clustering continuous-time document streams. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015a. find
  14. Du, N., Wang, Y., He, N., Sun, J., and Song, L. Time-sensitive recommendation from recurrent user activities. In Advances in Neural Information Processing Systems (NeurIPS), 2015b. find
  15. Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., and Song, L. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016. find
  16. Enguehard, J., Busbridge, D., Bozson, A., Woodcock, C., and Hammerla, N. Neural temporal point processes [for] modelling electronic health records. In Proceedings of Machine Learning Research, volume 136, pp. 85–113, 2020. find
  17. NeurIPS 2020 Workshop on Machine Learning for Health (ML4H). find
  18. Etesami, J., Kiyavash, N., Zhang, K., and Singhal, K. Learning network of multivariate Hawkes processes: A time series approach. arXiv preprint arXiv:1603.04319, 2016. — [arXiv:1603.04319](https://arxiv.org/abs/1603.04319) find
  19. Feng, X., Huang, L., Tang, D., Ji, H., Qin, B., and Liu, T. A language-independent neural network for event detection. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2016.
  20. Feng, Y., Zhou, B., Wang, H., Jin, H., and Roth, D. Generic temporal reasoning with differential analysis and explanation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2022. find
  21. Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760, 2022.
  22. Guo, F., Blundell, C., Wallach, H., and Heller, K. The Bayesian echo chamber: Modeling social influence via linguistic accommodation. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2015. find
  23. Han, R., Hsu, I.-H., Sun, J., Baylon, J., Ning, Q., Roth, D., and Peng, N. Ester: A machine reading comprehension dataset for reasoning about event semantic relations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. find
  24. Han, X. and Ning, Y. Text-enhanced multi-granularity temporal graph learning for event prediction. In Zhu, X., Ranka, S., Thai, M. T., Washio, T., and Wu, X. (eds.), IEEE International Conference on Data Mining, ICDM 2022, Orlando, FL, USA, November 28 - Dec. 1, 2022, pp. 171–180. find
  25. Hawkes, A. G. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 1971. find
  26. Hua, W., Mei, H., Zohar, S., Giral, M., and Xu, Y. Personalized dynamic treatment regimes in continuous time: A Bayesian joint model for optimizing clinical decisions with timing. Bayesian Analysis, 2022. find
  27. Ji, H. and Grishman, R. Refining event extraction through cross-document inference. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2008.
  28. Jianmo Ni, Jiacheng Li, J. M. Justifying recommendations using distantly-labeled reviews and fined-grained aspects. In Empirical Methods in Natural Language Processing (EMNLP), 2019. find
  29. Jin, X., Li, M., and Ji, H. Event schema induction with double graph autoencoders. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2022. find
  30. Jurafsky, D. and Martin, J. H. Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition. 2009. find
  31. Kassner, N., Tafjord, O., Sabharwal, A., Richardson, K., Schutze, H., and Clark, P. Language models with rationality. arXiv preprint arXiv:2305.14250, 2023. find
  32. Kazemi, S. M., Kim, N., Bhatia, D., Xu, X., and Ramachandran, D. Lambada: Backward chaining for automated reasoning in natural language. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
  33. Leetaru and Schrodt. Gdelt: Global data on events, location, and tone, 1979–2012. ISA annual convention, 2, 04 2013. find
  34. Lewis, P. A. and Shedler, G. S. Simulation of nonhomogeneous Poisson processes by thinning. Naval Research Logistics Quarterly, 1979. find
  35. Li, M., Zareian, A., Zeng, Q., Whitehead, S., Lu, D., Ji, H., and Chang, S.-F. Cross-media structured common space for multimedia event extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020. find
  36. Li, M., Li, S., Wang, Z., Huang, L., Cho, K., Ji, H., Han, J., and Voss, C. Future is not one-dimensional: Graph modeling based complex event schema induction for event prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021a. find
  37. Li, M., Ma, T., Yu, M., Wu, L., Gao, T., Ji, H., and McKeown, K. Timeline summarization based on event graph compression via time-aware optimal transport. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021b. find
  38. Li, Q., Ji, H., and Huang, L. Joint event extraction via structured prediction with global features. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2013.
  39. Li, S., Ji, H., and Han, J. Open relation and event type discovery with type abstraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. find
  40. Li, S., Zhao, R., Li, M., Ji, H., Callison-Burch, C., and Han, J. Opendomain hierarchical event schema induction by incremental prompting and verification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023. find
  41. Liniger, T. J. Multivariate Hawkes processes. Diss., Eidgenössische Technische Hochschule ETH Zürich, Nr. 18403, 2009. find
  42. Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., and Dustdar, S. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. find
  43. Lukasik, M., Srijith, P. K., Vu, D., Bontcheva, K., Zubiaga, A., and Cohn, T. Hawkes processes for continuous time sequence classification: An application to rumour stance classification in Twitter. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2016. find
  44. Lyu, Q., Zhang, H., Sulem, E., and Roth, D. Zero-shot event extraction via transfer learning: Challenges and insights. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
  45. Mei, H. and Eisner, J. The neural Hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems (NeurIPS), 2017a. find
  46. Mei, H. and Eisner, J. The neural hawkes process: A neurally self-modulating multivariate point process. In Proceedings of the International Conference on Machine Learning (ICML), 2017b. find
  47. Mei, H., Qin, G., Xu, M., and Eisner, J. Neural Datalog through time: Informed temporal modeling via logical specification. In Proceedings of the International Conference on Machine Learning (ICML), 2020. find
  48. Ning, Q., Feng, Z., Wu, H., and Roth, D. Joint reasoning for temporal and causal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018a. find
  49. Ning, Q., Wu, H., and Roth, D. A multi-axis annotation scheme for event temporal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018b. find
  50. Ning, Q., Wu, H., Han, R., Peng, N., Gardner, M., and Roth, D. Torque: A reading comprehension dataset of temporal ordering questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
  51. Omi, T., Ueda, N., and Aihara, K. Fully neural network based model for general temporal point processes. In Advances in Neural Information Processing Systems (NeurIPS), 2019. find
  52. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  53. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in PyTorch. 2017. find
  54. Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
  55. Russell, S. and Norvig, P. Artificial Intelligence: A Modern Approach. Prentice Hall, 2010. find
  56. Saravanakumar, K. K., Ballesteros, M., Chandrasekaran, M. K., and McKeown, K. Event-driven news stream clustering using entity-aware contextual embeddings. In Proceedings of the Conference of the European Chapter the Association for Computational Linguistics (EACL), 2021. find
  57. Sharma, K., Zhang, Y., Ferrara, E., and Liu, Y. Identifying coordinated accounts on social media through hidden influence and group behaviours. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2021. find
  58. Shchur, O., Biloš, M., and Günnemann, S. Intensity-free learning of temporal point processes. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. find
  59. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020. find
  60. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  61. Trivedi, R., Dai, H., Wang, Y., and Song, L. Know-Evolve: Deep temporal reasoning for dynamic knowledge graphs. In Proceedings of the International Conference on Machine Learning (ICML), 2017. find
  62. Trivedi, R., Farajtabar, M., Biswal, P., and Zha, H. DyRep: Learning representations over dynamic graphs. In Proceedings of the International Conference on Learning Representations (ICLR), 2019. find
  63. Wang, H., Chen, M., Zhang, H., and Roth, D. Joint constrained learning for event-event relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. find
  64. Wang, H., Zhang, H., Chen, M., and Roth, D. Learning constraints and descriptive segmentation for subevent detection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. find
  65. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions of Machine Learning Research (TMLR), 2022a.
  66. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E. H., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022b. — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  67. Weir, N. and Van Durme, B. Dynamic generation of interpretable inference rules in a neuro-symbolic expert system. arXiv preprint arXiv:2209.07662, 2022. — [arXiv:2209.07662](https://arxiv.org/abs/2209.07662) find
  68. Wen, H. and Ji, H. Utilizing relative event time to enhance event-event temporal relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. find
  69. Wen, H., Qu, Y., Ji, H., Ning, Q., Han, J., Sil, A., Tong, H., and Roth, D. Event time extraction and propagation via graph attention networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021. find
  70. Xiao, S., Yan, J., Farajtabar, M., Song, L., Yang, X., and Zha, H. Joint modeling of event sequence and time series with attentional twin recurrent neural networks. arXiv preprint arXiv:1703.08524, 2017. — [arXiv:1703.08524](https://arxiv.org/abs/1703.08524) find
  71. Xue, S., Shi, X., Zhang, Y. J., and Mei, H. Hypro: A hybridly normalized probabilistic model for long-horizon prediction of event sequences. In Advances in Neural Information Processing Systems (NeurIPS), 2022. find
  72. Xue, S., Shi, X., Chu, Z., Wang, Y., Zhou, F., Hao, H., Jiang, C., Pan, C., Xu, Y., Zhang, J. Y., et al. Easytpp: Towards open benchmarking the temporal point processes. arXiv preprint arXiv:2307.08097, 2023. — [arXiv:2307.08097](https://arxiv.org/abs/2307.08097) find
  73. Yang, C., Mei, H., and Eisner, J. Transformer embeddings of irregularly spaced events and their participants. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. find
  74. Ye, A., Cui, C., Shi, T., and Riedl, M. O. Neural story planning. arXiv preprint arXiv:2212.08718, 2022. — [arXiv:2212.08718](https://arxiv.org/abs/2212.08718) find
  75. Zhang, Q., Lipani, A., Kirnap, O., and Yilmaz, E. Self-attentive hawkes processes. In Proceedings of the International Conference on Machine Learning (ICML), 2020. find
  76. Zhang, T., Tham, I., Hou, Z., Ren, J., Zhou, L., Xu, H., Zhang, L., Martin, L. J., Dror, R., Li, S., et al. Human-in-the-loop schema induction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023a. find
  77. Zhang, Y., Cao, D., and Liu, Y. Counterfactual neural temporal point process for estimating causal influence of misinformation on social media. In Advances in Neural Information Processing Systems (NeurIPS), 2022. find
  78. Zhang, Z., Mei, H., and Xu, Y. Continuous-time decision transformer for healthcare applications. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2023b. find
  79. Zhou, B., Ning, Q., Khashabi, D., and Roth, D. Temporal common sense acquisition with minimal supervision. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020. find
  80. Zhou, B., Richardson, K., Ning, Q., Khot, T., Sabharwal, A., and Roth, D. Temporal reasoning on implicit events from distant supervision. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021. find
  81. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
  82. Zhu, S., Zhang, M., Ding, R., and Xie, Y. Deep fourier kernel for self-attentive point processes. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2021. find
  83. Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer hawkes process. In Advances in Neural Information Processing Systems (NeurIPS), 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2023-language-models-improve-event-prediction/