Quantifying and Predicting Disagreement in Graded Human Ratings

Paper · 2026 · Leixin Zhang and Çağrı Çöltekin

Summary

Annotators often disagree on inappropriate-language judgments, and the amount of disagreement differs from one item to the next. Zhang and Çöltekin ask whether a text alone reveals how much raters will diverge and whether they will split into opposing camps. They study offensive, hate speech and toxic language datasets that use Likert-scale ratings rather than binary labels, since graded scores preserve how spread out or polarized opinions are.

They compare two ways to estimate variance:

regressing the unbiased variance directly from item features, and
predicting the full rating distribution and computing its variance.

Because the scale is ordinal, training uses order-aware losses: Earth Mover's Distance (plus a variant) and cumulative cross-entropy. They also introduce the Opposition Index, a measure of how strongly ratings split into opposing stances.

The provided excerpt omits the quantitative results, so performance figures cannot be reported here. The work matters because predicting which items are likely to cause disagreement could guide how many annotators to assign, flag cases for expert review and support perspective-aware NLP.

Key points

Tasks: three detection problems (identifying offensiveness, hate speech and toxicity), each annotated with graded Likert ratings. The examples come from the Social Bias Frames offensive data (Sap et al., 2020), rated on a 0–2 scale.
Variance estimation: per-item unbiased variance is predicted either directly or from a predicted rating distribution.
Training losses: Earth Mover's Distance and a variant of it, plus cumulative cross-entropy.
Evaluation: MSE and Spearman's correlation against true variance, and F1 for detecting whether raters disagree at all.
Opposition Index: a new metric for bimodal, polarized ratings on Likert scales. The authors argue that standard bimodality tests, such as the Bimodality Coefficient, assume continuous data and fit ordinal ratings poorly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Institutions

🎓University of TübingenTübingen ·

Germany78 works · 41,280 citations

Fields, topics and concepts

📚 Shelves 6Computer Science · 18,614 works Artificial Intelligence · 17,965 works Arts and Humanities · 17,450 works Language and Linguistics · 17,296 works Social Sciences · 1,954 works General Social Sciences · 254 works

Cite

Cite as Leixin Zhang and Çağrı Çöltekin. 2026. Quantifying and Predicting Disagreement in Graded Human Ratings. Proceedings of the the fifth edition of NLPerspectives.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4qy8nuowzhpy

@inproceedings{zhang2026quantifying,
  title = {{Quantifying and Predicting Disagreement in Graded Human Ratings}},
  author = {Leixin Zhang and Çağrı Çöltekin},
  booktitle = {Proceedings of the the fifth edition of NLPerspectives},
  year = {2026},
  doi = {10.63317/4qy8nuowzhpy},
  url = {https://doi.org/10.63317/4qy8nuowzhpy}
}

References

5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.15896. — [arXiv:2106.15896](https://arxiv.org/abs/2106.15896) find
Jo Angouri. 2012. Managing disagreement in problem solving meeting talk. Journal of Pragmatics, 44(12):1565–1579. find
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, Alexandra Uma, et al. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future, pages 15–21. Association for Computationa…
Connor Baumler, Anna Sotnikova, and Hal Daumé III. 2023. Which examples should be multiply annotated? Active learning when annotators may disagree. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10352–10371, Toronto, Canada. Association for Computational Linguistics. find
Silvia Casola, Simona Frenda, Soda Marem Lo, Erhan Sezerer, Antonio Uva, Valerio Basile, Cristina Bosco, Alessandro Pedrani, Chiara Rubagotti, Viviana Patti, and Davide Bernardi. 2024. MultiPICo: Multilingual perspectivist irony corpus. In Proceedings of the 62nd Annual Meeting of the Association fo…

Updated 2 November 2026

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-quantifying-predicting-disagreement-graded-human/