Elamur Research⌕
☰
home / papers / 2025 / LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding

LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding

Paper · 2025 · Sheikh Jubair, Arwa Omayrah, Amal Alshammari et al.

Summary

LC-Eval is a benchmark for testing how well large language models understand very long inputs in English and Arabic. Its inputs range from about 4K tokens to more than 128K. The authors argue that existing long-context test sets mostly use multiple-choice or very short answers, seldom fill the full context window, and hardly cover Arabic. They also argue that word-overlap metrics such as BLEU and ROUGE, and plain similarity scores, misjudge open-ended answers.

The benchmark has four tasks:

  • multi-document QA, with deliberately similar distractor documents;
  • cross-lingual QA, where the question and the document are in different languages;
  • verifying each claim inside a paragraph;
  • multiple-choice questions over long texts.

GPT-4o drafted and self-graded the items. The drafts were then made harder in several stages, and three trained annotators checked them by majority vote. To score free-form answers, the authors propose entity-relationship recall, which uses an LLM as judge. Open and closed models, GPT-4o included, found the benchmark hard, and every model scored lower on Arabic than on English. This makes LC-Eval a rare public tool for long-context evaluation in Arabic.

Key points

  • Size and coverage: 7,903 samples split across eight subsets (four tasks × two languages). Length bins run from 4K–8K up to over 128K tokens, and the average word count reaches roughly 57K for English claim verification.
  • Sources:
    • 2024 Wikipedia dumps, WikiNews, WikiHow and WikiBooks in both languages;
    • Project Gutenberg books (English) and Hindawi books (Arabic);
    • Saudi Press Agency news. A dictionary-based filter removed harmful text.
  • Quality control:
    • GPT-4o rated each generated item at four temperatures (0.0, 0.3, 0.6, 0.9).
    • An item was kept only if accuracy, grammar, cultural sensitivity and safety all scored ≥ 9.
    • 1,300 multi-document items then went to human validation.
  • Memorization check: GPT-4o averaged 75.88 with the context and 56.41 without it, a gap of about 20 points. The gap was larger for Arabic.
  • Human check of the metric: On 50 multi-document samples per model, human judgments of relation presence agreed well with the automatic entity-relationship recall on which models came out best.

Interesting ideas

  • I1. Score free answers by whether the gold answer's entity relations appear in the response, not by word overlap. Overlap metrics would rate "capital is Paris" and "capital is Rome" as near-identical despite opposite meanings.
  • I2. Run the same questions with the context removed, to measure how much a model answers from prior knowledge instead of from the text.
  • I3. Build multi-document inputs from MinHash-similar distractors, and ask the model to name which documents its answer came from. This exposes answers that look right but cite the wrong sources.
  • I4. Accept generated items only when the generator's own grades stay high and consistent across several sampling temperatures. This works like an agreement filter over multiple runs.
  • I5. Draw the text chunk for a question from the middle of a document most of the time, cut at sentence boundaries, so that questions do not cluster at the opening.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Test answers could be graded by matching relation triples from the answer's meaning graph against a gold graph. This gives an explainable, paraphrase-proof score for fable QA and school tests.
  • I2 (learn): This makes a natural negative control. Readers should fail when the story is absent, and any success points to leakage from compiled knowledge.
  • I3 (world): Mix near-duplicate stories into one input to stress-test whether entity and event tracking stays separated per document. Also require each answer to point back to the source text that supports it.
  • I4 (learn): LLM teacher tiers could be sampled at several temperatures, keeping only labels that stay stable. This mirrors the agreement of several witness parsers on silver data.
  • I5 (answer): QA sets built over long tales could use the same middle-weighted sampling. That checks whether the world model holds up deep into a text, not just near its start.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Saud UniversityRiyadh · Saudi Arabia38 works · 527 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Sheikh Jubair, Arwa Omayrah, Amal Alshammari, Alhanoof Althnian, Abdulhamed Alothaimen, Norah A. Alzahrani, Shahad D. Alzaidi, Nora Al-Twairesh and Abdulmohsen Al-Thubaity. 2025. LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding. Findings of the Association for Computational Linguistics: EMNLP 2025.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-emnlp.1057

@inproceedings{jubair2025lc,
  title = {{LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding}},
  author = {Sheikh Jubair, Arwa Omayrah, Amal Alshammari, Alhanoof Althnian, Abdulhamed Alothaimen, Norah A. Alzahrani, Shahad D. Alzaidi, Nora Al-Twairesh and Abdulmohsen Al-Thubaity},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2025},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.1057},
  url = {https://doi.org/10.18653/v1/2025.findings-emnlp.1057}
}

References

31 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, et al. 2025. Fanar: An arabic-centric multimodal generative ai platform. arXiv preprint arXiv:2501.13944. ↗
  2. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219. ↗
  3. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. Preprint, arXiv:2307.11088. ↗
  4. Anthropic. Introducing Claude 3.5 Sonnet — anthropic.com. https://www.anthropic.com/ [Accessed 29-01-news/claude-3-5-sonnet. 2025]. ↗
  5. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. Preprint, arXiv:2308.14508. ↗
  6. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. Preprint, arXiv:2412.15204. ↗
  7. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2024. Allam: Large language models for arabic and english. Preprint, arXiv:2407.15390. ↗
  8. Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, and Timofey Bryksin. 2024. Long code arena: a set of benchmarks for long-context code models. Preprint, arXiv:2406.11612. ↗
  9. Cohere. Cohere’s Command R+ Model (Details and Application) — Cohere — docs.cohere.com. https://docs.cohere.com/v2/docs/command-r-plus. [Accessed 29-01-2025]. ↗
  10. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. ↗
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  12. Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, page 166–175, New York, NY, USA. Association for Computing Machinery. find
  13. Project Gutenberg. Project Gutenberg — gutenberg.org. https://www.gutenberg.org/. [Accessed 29-01-2025]. ↗
  14. Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. 2024. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models. Preprint, arXiv:2408.10151. ↗
  15. Hindawi. hindawi.org. https://www.hindawi.org/. [Accessed 29-01-2025]. ↗
  16. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? Preprint, arXiv:2404.06654. ↗
  17. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  18. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A "novel" challenge for long-context language models. Preprint, arXiv:2406.16264. ↗
  19. Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, and Kelvin Guu. 2024. Can long-context language models subsume retrieval, rag, sql, and more? Preprint, arXiv:2406.13121. ↗
  20. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  21. OpenAI. Models. https://platform.openai.com/ docs/models. [Accessed 29-01-2025]. ↗
  22. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. 2022. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5336–5358, Seattle, United States. Association for Computational Linguistics. ↗
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  24. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. ↗
  25. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149. ↗
  26. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics.
  27. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel Bowman. 2022. Squality: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1139–1156. find
  28. Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024. Leave no document behind: Benchmarking long-context LLMs with extended multi-doc QA. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5627–5646, Miami, Florida, USA. Association for Computational Linguistics. ↗
  29. WorldData.info. 2024. Arabic - Worldwide distribution — worlddata.info. https://www.worlddata.info/ languages/arabic.php. [Accessed 13-02-2025]. ↗
  30. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics. ↗
  31. Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Mosen Alharthi, Bang An, Juncai He, Xiangbo Wu, Fei Yu, Junying Chen, Zhuoheng Ma, Yuhao Du, Yan Hu, He Zhang, Emad A. Alghamdi, Lian Zhang, Ruoyu Sun, Haizhou Li, Benyou Wang, and Jinchao Xu. 2024. Second language (arabic) acquisition of llms via progressive vocabulary expansion. arXiv preprint arXiv:2412.12310. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jubair-2025-lc-eval-bilingual-multi-task/