📄 Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring

Summary

The paper addresses how to estimate how hard a question is for a large language model, as opposed to how hard it is for human readers or retrieval systems. The authors propose Q-DAPS. An LLM (LLaMA 3.3 by default) generates up to 20 plausible but wrong candidate answers, each with a plausibility score and a justification. The candidates are validated for duplicates (using BEM), score range and count. Their scores are then down-weighted by popularity, measured with Wikipedia page views, and the Shannon entropy of the resulting distribution is normalized to a 0–1 difficulty value. The intuition is that a question is harder when many wrong answers look about equally convincing.

The method was tested on TriviaQA, NQ, MuSiQue and QASC. It separated easy from hard questions better than readability, popularity (PopQA), retrieval-complexity, prompt-based and uncertainty-based baselines, and its estimates agreed with human judgments. The approach is interpretable, because the candidates and their scores are returned alongside the difficulty value. The authors suggest it can be used for hallucination-risk detection, model selection and question routing.

Key points

• Debiasing formula: plausibility × (1 − α × popularity), using Wikipedia page views from 2015 to 2024. Candidates with no Wikipedia page get zero popularity.
• Validation loop: if a generated candidate list fails validation, generation is repeated with the temperature raised by 0.1 each time.
• Main results: measured by Cohen's d, Q-DAPS with LLaMA 3.3 70B reaches roughly 0.75–1.09 across the four datasets. No baseline in that comparison exceeds about 0.48.
• Ablations: Q-DAPS still beats the baselines when the gold answer is withheld, when debiasing is removed, and when smaller cores are used (Qwen 2.5 7B, LLaMA 3.1 8B). Debiasing gives a statistically significant gain (paired t-test, p = 0.0356).
• Resources and future work: code is released at github.com/DataScienceUIBK/Q-DAPS. The authors list multilingual and multimodal extensions as future work.