Elamur Research⌕
☰
home / papers / 2026 / Findings of the MAGMaR 2026 Shared Task

Findings of the MAGMaR 2026 Shared Task

Paper · 2026 · Alexander Martin, Dengjia Zhang, Joel Brogan et al.

Summary

This overview reports the MAGMaR 2026 shared task, held at the second MAGMaR workshop at ACL. The task addresses multimodal information seeking, an area where most benchmarks still focus on text alone. Participants either ranked videos for a persona-specific query about a real 2025 event, or wrote cited articles grounded in relevant videos. The test collection merges WikiVideo25 with MultiVENT 2.0 distractor videos, giving over 110,000 multilingual clips. WikiVideo25 extends WikiVideo with persona queries, more raw bystander footage, and Mandarin and Nepali–English topics. In retrieval, both teams beat all baselines. The strongest approaches combined broad first-stage recall with reasoning-based reranking, and text proxies of videos worked better than video-native embeddings. In generation, iterative question-answering pipelines were rated best by human judges. However, the automatic MiRAGE metric scored those systems lower because it treats justified abstention as lost information. The authors argue that future evaluation should credit appropriate refusal and that models should reason over visual content directly.

Key points

  • Retrieval track: 2 teams submitted 17 runs, evaluated with nDCG and Recall at 10, 20 and 100. C2F-RAG ranked first (nDCG@10 0.848) by indexing summaries and keyframe captions with BGE-M3 and reranking with an LLM agent. MARQUIS-2 reached 0.759.
  • Retrieval baselines: OmniEmbed scored 0.166, OmniEmbed + RankVideo 0.542, and the closed-source Wholembed-v3 0.717. Reranking alone more than tripled the weak dense baseline.
  • Generation track: 4 teams submitted 16 systems for 19 queries in an oracle setting, where systems received the ground-truth relevant videos. Outputs were judged by 3 human annotators and by MiRAGE, which uses CLUE for entailment judgments.
  • Top generation systems: MARQUIS-iter-qa-base (3.833) and TRACE-04 (3.825) received the best human scores, while the CAG baseline with Qwen3.5 scored 3.088. Iterative QA clearly outperformed single-shot QA (3.833 vs. 3.070).
  • Metric disagreement: systems with the highest automatic precision, CAG and MARQUIS-Ginger (InfoP 0.764 and 0.776), ranked 6th or lower with human judges. No team used task-specific fine-tuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🏢OpenAISan Francisco · United States18 works · 27,146 citations🎓Huazhong University of Science and TechnologyWuhan · China45 works · 6,345 citations🎓University of Maryland, Baltimore CountyBaltimore · United States14 works · 915 citations🔬United States Air Force Research LaboratoryWright-Patterson AFB · United States2 works · 185 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Alexander Martin, Dengjia Zhang, Joel Brogan, Francis Ferraro, Jeremy Gwinnup, Reno Kriz, Teng Long, Kenton Murray, Andrew Yates and Xiang Xiang. 2026. Findings of the MAGMaR 2026 Shared Task. Proceedings of the 2nd Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.magmar-main.16

@inproceedings{martin2026findings,
  title = {{Findings of the MAGMaR 2026 Shared Task}},
  author = {Alexander Martin, Dengjia Zhang, Joel Brogan, Francis Ferraro, Jeremy Gwinnup, Reno Kriz, Teng Long, Kenton Murray, Andrew Yates and Xiang Xiang},
  booktitle = {Proceedings of the 2nd Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.magmar-main.16},
  url = {https://doi.org/10.18653/v1/2026.magmar-main.16}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mixedbread Team. 2026. Beyond the limit: Introduce mixedbread wholembed v3. Blog post. find
  2. Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, Pengyu Yan, Akhil Gorugantu, and David Doermann. 2026. Craft: Critic-refined adaptive key-frame targeting for multimodal video question answering. Preprint, arXiv:2605.19075. find
  3. Debashish Chakraborty, Dengjia Zhang, Jialiang Jin, Hanting Liu, Katherine Guerrerio, Hanxiang Qin, Tyler Skow, Alexander Martin, Reno Kriz, and Benjamin Van Durme. 2026. Marquis: A three-stage pipeline for video retrieval-augmented generation. Preprint, arXiv:2605.17640. find
  4. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2025. M3-embedding: Multilinguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.
  5. Jiaxin Dai, Zehang Wei, Jiamin Yan, and Xiang Xiang. 2026. Decoupling semantics and logic: A training-free coarse-to-fine pipeline for video retrieval-augmented generation. Arxiv preprint, 2606.07924. find
  6. Norway, April 10-14, 2022, Proceedings, Part II, volume 13186 of Lecture Notes in Computer Science, pages 305–310. Springer. find
  7. Alexander Martin, William Walden, Reno Kriz, Dengjia Zhang, Kate Sanders, Eugene Yang, Chihsheng Jin, and Benjamin Van Durme. 2026. Seeing through the mirage: Evaluating multimodal retrieval augmented generation. Preprint, arXiv:2510.24870. — [arXiv:2510.24870](https://arxiv.org/abs/2510.24870) find
  8. Qwen Team. 2026. Qwen3.5: Towards native multimodal agents. find
  9. Tyler Skow, Alexander Martin, Benjamin Van Durme, Rama Chellappa, and Reno Kriz. 2026. Rankvideo: Reasoning reranking for text-to-video retrieval. Preprint, arXiv:2602.02444. — [arXiv:2602.02444](https://arxiv.org/abs/2602.02444) find
  10. Pengyu Yan, Akhil Gorugantu, Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, and David Doermann. 2026. Trace: Evidence grounding-guided multi-video event understanding and claim generation. Preprint, arXiv:2605.16740.
  11. Dengjia Zhang, Alexander Martin, William Jurayj, Kenton Murray, Benjamin Van Durme, and Reno Kriz. 2026. Unified multimodal uncertain inference. Preprint, arXiv:2604.08701. — [arXiv:2604.08701](https://arxiv.org/abs/2604.08701) find
  12. A Reno Kriz, Kate Sanders, David Etter, Kenton Murray, Cameron Carpenter, Kelly Van Ochten, Hannah Recknor, Jimena Guallar-Blasco, Alexander Martin, Ronald Colaianni, Nolan King, Eugene Yang, and Benjamin Van Durme. 2025. Multivent 2.0: A massive multilingual benchmark for event-centric video retrie… — [arXiv:2410.11619](https://arxiv.org/abs/2410.11619) find
  13. Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, and Eugene Yang. 2025. Overview of the trec 2024 neuclir track. Preprint, arXiv:2509.14355. — [arXiv:2509.14355](https://arxiv.org/abs/2509.14355) find
  14. Xueguang Ma, Luyu Gao, Shengyao Zhuang, Jiaqi Samantha Zhan, Jamie Callan, and Jimmy Lin. 2025. Tevatron 2.0: Unified document retrieval toolkit across scale, language, and modality. Preprint, arXiv:2505.02466. — [arXiv:2505.02466](https://arxiv.org/abs/2505.02466) find
  15. Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2022. Streamlining evaluation with ir-measures. In Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, W IKI V IDEO 25 Dataset find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/martin-2026-findings-magmar-2026-shared-task/