Elamur Research⌕
☰
home / papers / 2026 / Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction

Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction

Paper · 2026 · Vipul Kumar Rathore, Malik Hammad Faisal, Parag Singla et al.

Summary

Distantly supervised relation extraction (DSRE) trains on noisy, bag-level labels obtained by aligning text with knowledge-base triples, yet it must make predictions for individual sentences. Fine-tuned DSRE models therefore tend to mix up closely related relations. LLMs prompted directly also struggle, because the noisy data yields poor in-context examples.

The authors propose HYDRE, a hybrid pipeline with three stages:

  • A trained DSRE model (PARE) narrows the label space to the top-k candidate relations.
  • For each candidate, a training bag is retrieved using a weighted mix of semantic similarity to the query and PARE confidence.
  • From each bag, the sentence covering the most confidently predicted bag labels is chosen as the exemplar. The LLM then picks the final relation(s).

The framework is extended to cross-lingual RE. For this, the authors built a new gold benchmark for four low-resource Indic languages and tested three transfer settings.

HYDRE gains up to 20 F1 points in English and about 17 F1 on average for the Indic languages over prior DSRE models and naive prompting. This shows that a specialised model and an LLM work well together in noisy and low-resource settings.

Key points

  • The new evaluation benchmark covers Oriya, Santali, Manipuri and Tulu. The transfer settings are English-only, translate-train and translate-test. Code is released at github.com/dair-iitd/HYDRE.
  • PARE reaches roughly 85% (English) and 70% (Indic) sentence-level Recall@5, which motivates using it as a candidate generator. The authors fix k = 5.
  • Ablations show that dropping retrieval components costs up to 7 F1 in English and 12 F1 cross-lingually.
  • On Wiki-20m (81 relations), HYDRE beats zero-shot prompting by up to 36 micro-F1 points for Llama-3.1 and Qwen3. For Llama-3.1, micro-F1 rises from 13 to 49.
  • Error analysis on NYT-10m identifies three main problems:
    • Recency bias: 34% of errors simply repeat PARE's top-ranked candidate.
    • Missing gold relations: the correct relation is absent from the top-5 candidates for about 14% of queries.
    • Inherently ambiguous relation pairs, such as nationality vs. ethnicity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology DelhiNew Delhi · India22 works · 1,036 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksInformation extraction · 167 works

Cite

Cite as Vipul Kumar Rathore, Malik Hammad Faisal, Parag Singla and Mausam. 2026. Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2109

@inproceedings{rathore2026combining,
  title = {{Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction}},
  author = {Vipul Kumar Rathore, Malik Hammad Faisal, Parag Singla and Mausam},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2109},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2109}
}

References

35 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, E. Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
  2. Abhyuday Bhartiya, Kartikeya Badola, and Mausam. 2022. DiS-ReX: A multilingual dataset for distantly supervised relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland. Association for Computational Linguistics. find
  3. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. find
  4. Tao Chen, Haizhou Shi, Siliang Tang, Zhigang Chen, Fei Wu, and Yueting Zhuang. 2021. CIL: Contrastive instance learning framework for distantly supervised relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… find
  5. Yang Chen, Chao Jiang, Alan Ritter, and Wei Xu. 2023. Frustratingly easy label projection for cross-lingual transfer. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5775–5796. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  7. Jay Gala, Pranjal A Chitale, A K Raghavan, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar M, Janki Atul Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. Indictrans2: Towards high-quality and accessible machine tran…
  8. Tianyu Gao, Xu Han, Yuzhuo Bai, Keyue Qiu, Zhiyu Xie, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Manual evaluation matters: Reviewing test protocols of distantly supervised relation extraction. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pag… find
  9. Shivanshu Gupta, Matt Gardner, and Sameer Singh. 2023. Coverage-based example selection for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13924–13950, Singapore. Association for Computational Linguistics. find
  10. Zhaorui Jian, Shengquan Liu, Wei Gao, and Jianming Cheng. 2024. Distantly supervised relation extraction based on non-taxonomic relation and self-optimization. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE. find
  11. Janak Kapuriya, Manit Kaushik, Debasis Ganguly, and Sumit Bhatia. 2025. Exploring the role of diversity in example selection for in-context learning. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’25, page 2962–2966, New Yor… find
  12. Duong Minh Le, Yang Chen, Alan Ritter, and Wei Xu. 2024. Constrained decoding for cross-lingual label projection. In The Twelfth International Conference on Learning Representations. find
  13. Dongyang Li, Taolin Zhang, Nan Hu, Chengyu Wang, and Xiaofeng He. 2022. Hiclre: A hierarchical contrastive learning framework for distantly supervised relation extraction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2567– 2578. find
  14. Junpeng Li, Zixia Jia, and Zilong Zheng. 2023a. Semi-automatic data enhancement for document-level relation extraction with distant supervision from large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5495–5505, Singapore. Associati… find
  15. Min Li, Cong Shao, Gang Li, and Mingle Zhou. 2023b. Hfmre: Constructing huffman tree in bags to find excellent instances for distantly supervised relation extraction. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12820– 12832. find
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  17. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. find
  18. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 10…
  19. Tom M. Mitchell, Justin Betteridge, Andrew Carlson, Estevam R. Hruschka Jr., and Richard C. Wang. 2009. Populating the semantic web by macro-reading internet text. In The Semantic Web - ISWC 2009, 8th International Semantic Web Conference, ISWC 2009, Chantilly, VA, USA, October 25-29, 2009. Proceedi… find
  20. Arijit Nag, Soumen Chakrabarti, Animesh Mukherjee, and Niloy Ganguly. 2025. Efficient continual pre-training of LLMs for low-resource languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  21. Arijit Nag, Bidisha Samanta, Animesh Mukherjee, Niloy Ganguly, and Soumen Chakrabarti. 2021. A data bootstrapping recipe for low-resource multilingual relation classification. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 575–587. find
  22. Jian Ni, Taesun Moon, Parul Awasthy, and Radu Florian. 2020. Cross-lingual relation extraction with transformers. arXiv preprint arXiv:2010.08652. — [arXiv:2010.08652](https://arxiv.org/abs/2010.08652) find
  23. Vipul Rathore, Kartikeya Badola, Parag Singla, et al. 2022. Pare: A simple and strong baseline for monolingual and multilingual distantly supervised relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 340–354… find
  24. Vipul Kumar Rathore, Aniruddha Deb, Ankish Kumar Chandresh, Parag Singla, and Mausam . 2024. SSP: Self-supervised prompting for cross-lingual transfer to low-resource languages using large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15081–15102, M… find
  25. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. IndicGen-Bench: A multilingual benchmark to evaluate generation capabilities of LLMs on Indic languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  26. Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D. Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 455–46… find
  27. Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Talukdar. 2018. RESIDE: Improving distantly-supervised neural relation extraction using side information. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1257–126… find
  28. Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2023. Revisiting relation extraction in the era of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15566– 15589. find
  29. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
  30. Peng Wang, Xiaobin Wang, Chao Lou, Shengyu Mao, Pengjun Xie, and Yong Jiang. 2024. Effective demonstration annotation for in-context learning via language model-based determinantal point process. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1266–12… find
  31. Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1753–1762. find
  32. Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35–45, Copenhagen, Denmark. Association for Compu… find
  33. Xiaoyan Zhao, Yang Deng, Min Yang, Lingzhi Wang, Rui Zhang, Hong Cheng, Wai Lam, Ying Shen, and Ruifeng Xu. 2024. A comprehensive survey on relation extraction: Recent advances and new frontiers. ACM Computing Surveys, 56(11):1–39. find
  34. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  35. Sizhe Zhou, Yu Meng, Bowen Jin, and Jiawei Han. 2024. Grasping the essentials: Tailoring large language models for zero-shot relation extraction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13462–13486, Miami, Florida, USA. Association for Computa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rathore-2026-combining-distantly-supervised-models-context/