Argument-Based Comparative Question Answering Evaluation Benchmark
Paper · 2026 · Irina Nikishina, Saba Anwar, Nikolay Dolgov et al.
Summary
Large language models can produce fluent answers to comparative questions such as "Simpsons or Family Guy?". Judging those answers is still difficult, because there are no agreed standards for comparative question answering (CQA) and manual annotation is expensive. The authors address this with CompQA, an LLM-as-a-Judge benchmark. It scores comparative summaries on 15 criteria grouped into structure, relevance and quality, giving a total of 0 to 19 points.
They built evaluation sets of object pairs with optional aspects and supporting arguments, drawing on CAM 2.0, Yahoo! Answers and earlier CQA data. Several LLMs then generated answers under four prompting scenarios, and both human annotators and LLM judges rated the results. Automatic and human scores largely agreed, although the LLM judge was somewhat more generous. Proprietary systems led the ranking. However, a small open model fine-tuned on top-rated answers reached comparable quality.
The benchmark matters because it gives CQA research a reproducible, low-cost evaluation protocol. All data, code and evaluation results are publicly released.
Key points
- Scoring scheme. CompQA uses 15 criteria worth up to 19 points: structure accounts for 7, relevance for 5, and the rest covers quality aspects such as conciseness, factuality, informativeness and coherence. The judge LLM returns per-criterion scores as JSON and does not compute a total.
- Datasets. There is a main set of 50 comparative questions and an extended set of 432. In the main set, 20 pairs have a defined aspect, and each pair has 3–20 arguments (10 on average).
- Models evaluated.
- Main experiments: GPT-3.5, GPT-4/GPT-4o, Llama-3-8B-Instruct, Llama-3-70B-Instruct, Perplexity and Mixtral.
- Additional runs: Gemma-3-27B, Qwen3-8B, DeepSeek-R1 and Gemini-2.5-Flash.
- GPT-4o and Perplexity scored highest.
- Judge reliability. Llama-3-70B used as a judge broadly matched human ratings but tended to overrate answers that humans scored low.
- Fine-tuning result. Llama-3-8B-Instruct was fine-tuned with LoRA on 1,244 answers that had scored ≥18. It reached 18.40 ± 1.25, roughly on par with GPT-4o. The resources are available on GitHub at
lilaspourpre/cqa-evaluation-framework.
Fields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksQuestion answering · 441 works
Cite
Cite as Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin and Chris Biemann. 2026. Argument-Based Comparative Question Answering Evaluation Benchmark. Proceedings of the 13th Workshop on Argument Mining and Reasoning.
@inproceedings{nikishina2026argument,
title = {{Argument-Based Comparative Question Answering Evaluation Benchmark}},
author = {Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin and Chris Biemann},
booktitle = {Proceedings of the 13th Workshop on Argument Mining and Reasoning},
year = {2026},
doi = {10.18653/v1/2026.argmining-1.6},
url = {https://doi.org/10.18653/v1/2026.argmining-1.6}
}References
- Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya… — LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/nikishina-2026-argument-based-comparative-question-answering/