Elamur Research⌕
☰
home / papers / 2026 / AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic

AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic

Summary

Static Arabic benchmarks are slow and costly to build, often need human annotators, and gradually go stale or leak into training data. Existing dynamic benchmarks are mostly English-only and tied to one knowledge source. AraDynFact is an automatic pipeline that turns any Arabic document collection into a fresh test of factual knowledge. Documents are split into paragraphs and sentences. Each sentence is broken into atomic facts, salient entities are tagged, and the facts are rewritten to stand alone and then trimmed. Questions are generated from a document's fact list and labelled by type and difficulty. Models answer with no context, and an LLM judge grades each reply against the source facts instead of a fixed gold answer. Every stage runs on Qwen3-235B-A22B with task-specific prompts. The authors applied it to Arabic Wikipedia and tested 17 models. Large general models led, and Arabic-centric models beat same-size general ones only partly. Scores rose steadily with Qwen3 model size and showed strong Spearman rank agreement with established Arabic suites. The result is a cheap, regenerable, leakage-resistant probe of knowledge coverage.

Key points

  • Pipeline steps: decomposition, atomic fact extraction, NER, decontextualization, refinement, question generation, answering and LLM-as-a-Judge grading, all done with Qwen3-235B-A22B.
  • Question taxonomy and difficulty:
    • Four types: Factoid, Explanatory, Causal Why/How, Comparative.
    • Three difficulty levels: low, medium, high.
    • Factoid questions are mostly low because they usually rest on a single fact.
  • Grading labels:
    • INCORRECT covers partial answers, missing key facts or contradictions.
    • NOT_ATTEMPTED covers stated uncertainty or replies that add nothing beyond the question.
  • Model results:
    • GPT-OSS-120B, DeepSeek-R1-Distill-Llama-70B, Llama-3.3-70B and Gemma-3-27B scored highest.
    • SILMA-9B, Llama-3.1-8B and DeepSeek-R1-Distill-Llama-8B lagged.
    • High-difficulty scores were unstable, with some near zero, likely because there are few such items.
  • Validation: monotonic gains across Qwen3 sizes, plus strong Spearman correlation with ArabicMMLU, Exams, MedinahQA, AraTrust (from OALL), 3LM and ArabCulture.

Interesting ideas

  • I1. Grade an answer against the set of facts the question was built from, not against one reference string; this avoids penalising correct but differently worded answers. — For Mova (answer): QA on fables and tales could generate questions from facts in the world of the text and check a reader's reply against exactly those facts.
  • I2. Rewrite extracted facts so each is self-contained, then run a separate pass to keep them short; this makes facts portable and independently checkable. — For Mova (frames): after coreference, each frame could be emitted as a standalone proposition with pronouns resolved, giving clean units for the world layer.
  • I3. Set question difficulty by how many facts must be combined to answer it, making difficulty a measurable property rather than a guess. — For Mova (answer): school-course tests and word problems could be graded by the number of world facts or inference steps an answer requires.
  • I4. Check a benchmark's validity by testing whether it ranks same-family models of growing size in order; if it does not, the benchmark is insensitive. — For Mova (learn): run evaluations on deliberately weakened or smaller Mova configurations as an ordering sanity check, alongside negative controls.
  • I5. Keep abstentions separate from errors, and count replies that merely restate the question as abstentions rather than successes. — For Mova (answer): readers should report "unknown" as its own outcome and not get credit for echoing the query, so explainable confidence stays honest.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.35461 · arXiv 2609.35461

@misc{iacobacci2026aradynfact,
  title = {{AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic}},
  author = {Iacobacci, Ignacio and Altam, Faroq and Qian, Zhaozhi and Alqurishi, Muhammad},
  year = {2026},
  doi = {10.48550/arxiv.2609.35461},
  eprint = {2609.35461},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.35461}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Harethah Abu Shairah, Somayah AlHarbi, Abdulaziz AlHussein, Sameer Alsabea, Omar Shaqaqi, Hebah AlShamlan, Omar Knio, and George Turkiyyah. 2025. ALARB: An Arabic legal argument reasoning benchmark. In Proceedings of The Third Arabic Natural Language Processing Conference, pages 389–406, Suzhou, China. find
  2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. ↗ our card
  3. Emad A Alghamdi, Reem Masoud, Deema Alnuhait, Afnan Y Alomairi, Ahmed Ashraf, and Mohamed Zaytoon. 2025. Aratrust: An evaluation of trustworthiness for llms in arabic. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8664–8679. find
  4. Yazeed Alnumay, Alexandre Barbet, Anna Bialas, William Darling, Shaan Desai, Joan Devassy, Kyle Duffy, Stephanie Howe, Olivia Lasche, Justin Lee, Anirudh Shrinivason, and Jennifer Tracey. 2025. Command r7b arabic: A small, enterprise focused, multilingual, and culturally aware arabic llm. Preprint, arXiv:2503.14603. ↗
  5. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, Abdelrahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-Chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, and 25 others. 2025. Palm: A culturally inclusive and linguistically diverse dataset for arabic llms. Preprint, arXiv:2503.00151. ↗
  6. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Dr. Majed Alrubaian, Ali Alammari, Zaki Alawami, and 7 others. 2025. AL-Lam: Large language models for arabic and english. In The Thirteenth International Conference on Learning Representations. find
  7. Basma El Amel Boussaha, Leen AlQadi, Mugariya Farooq, Shaikha Alsuwaidi, Giulia Campesan, Ahmed Alzubaidi, Mohammed Alyafeai, and Hakim Hacid. 2025. 3lm: Bridging arabic, stem, and code through benchmarking. arXiv preprint arXiv:2507.15850. ↗
  8. Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and 1 others. 2025. Recent advances in large langauge model benchmarks against data contamination: From static to dynamic evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  9. Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. 2021. Decontextualization: Making sentences stand-alone. Transactions of the Association for Computational Linguistics, 9:447–461. find
  10. Ali Elfilali, Hamza Alobeidli, Clémentine Fourrier, Basma El Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. 2024. Open arabic llm leaderboard. https://huggingface.co/ spaces/OALL/Open-Arabic-LLM-Leaderboard. ↗
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  12. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. ↗ our card
  13. Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, and Xiangliang Zhang. 2025. Datagen: Unified synthetic dataset generation via large language models. In The Thirteenth International Conference on Learning Representations. find
  14. Meta AI. 2026. The llama 4 herd: Architecture, training, evaluation, and deployment notes. ArXiv, abs/2601.11659. ↗
  15. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. Preprint, arXiv:2305.14251. ↗ our card
  16. Navid-AI. 2025. Yehia 7b preview. https://huggingface.co/Navid-AI/Yehia-7B-preview. ↗
  17. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗ our card
  18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find
  19. Abdelrahman Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Cahtrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, and Fajri Koto. 2025. Commonsense reasoning in arab culture. Preprint, arXiv:2502.12788. silma-ai. 2024. Silma 9b instruct https://huggingface.co/silma-ai/ SILMA-9B-Instruct-v1.0. v1.0. ↗
  20. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, and 23 others. Fanar: An arabic-centric multimodal generative ai platform. Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, and 1 others. 2024. Factcheck-bench: Fine-grained evaluation benchmark for automatic fact-checkers. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14199–14230. find our card
  21. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. Preprint, arXiv:2411.04368. ↗ our card
  22. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. Livebench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations. find
  23. Jiahao Ying, Yixin Cao, Yushi Bai, Qianru Sun, Bo Wang, Wei Tang, Zhaojun Ding, Yizhe Yang, Xuanjing Huang, and Shuicheng YAN. 2024. Automating dataset updates towards reliable and timely evaluation of large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/iacobacci-2026-aradynfact-dynamic-evaluation-factual-knowledge/