Elamur Research
☰
home / papers / 2021 / Toward a Perspectivist Turn in Ground Truthing for Predictive Computing

Toward a Perspectivist Turn in Ground Truthing for Predictive Computing

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2109.04270

Summary

Supervised machine learning usually relies on datasets in which several annotators' labels are merged, typically by majority vote, into a single "gold standard." The authors argue that this aggregation discards useful information. It can silence minority experts in hard cases and erase legitimate disagreement on ambiguous items, which are common in NLP. They contend the problem also affects tasks usually treated as objective, such as clinical decision support. This position paper presents no new experiments. Instead, it proposes data perspectivism, a paradigm that keeps raters' differing judgments in the ML pipeline rather than collapsing them. It defines a weak and a strong form of the idea and models the ground-truthing workflow as a BPMN diagram. The authors survey ways to train and evaluate models on non-aggregated labels and weigh the benefits against the drawbacks. They close with recommendations and a ten-point research agenda. The paper matters because it shifts attention from data quality alone to representativeness, fairness and explainability. It invites practitioners to ask whose opinions a model actually encodes.

Key points

  • Weak perspectivism collects many labels per object, producing a "diamond standard," but then reduces them to one label, for example by weighted voting ("perspective reduction"). Strong perspectivism keeps all labels through both training and benchmarking.
  • Raters may disagree for reasons other than error, including:
    • ambiguity of the phenomenon
    • instability of the phenomenon over time
    • task complexity
    • cognitive biases
    • "decision noise" in Kahneman's sense
  • Proposed implementation strategies include:
    • replicating training objects once for each label they received (the most backward-compatible option)
    • ensemble methods
    • soft-loss training
  • Suggested evaluation approaches include testing on items with "overwhelming majority" agreement and using soft metrics. The authors call for unified metrics that balance richness of perspectives against reliability, for instance by combining inter-rater agreement with entropy-based measures.
  • The research agenda also covers:
    • fairness, including detecting biases introduced by groups of raters
    • explainable AI
    • simulating observer variability
    • richer rating formats, such as rankings and confidence levels, drawing on computational social choice

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

140 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 137 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1609/aaai.v37i6.25840 · arXiv 2109.04270

@misc{basile2021toward,
  title = {{Toward a Perspectivist Turn in Ground Truthing for Predictive Computing}},
  author = {Basile, Valerio and Cabitza, Federico and Campagner, Andrea and Fell, Michael},
  year = {2021},
  doi = {10.1609/aaai.v37i6.25840},
  eprint = {2109.04270},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1609/aaai.v37i6.25840}
}

References

13 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Akhtar, S., Basile, V., Patti, V.: A new measure of polarization in the annotation of hate speech. In: Alviano, M., Greco, G., Scarcello, F. (eds.) AI*IA 2019 – Advances in Artificial Intelligence. pp. 588–603. find
  2. Springer International Publishing, Cham (2019) 2. Akhtar, S., Basile, V., Patti, V.: Modeling annotator perspective and polarized opinions to improve hate speech detection. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. vol. 8, pp. 151–154 (2020) 3. Aroyo, L., Welty,… find
  3. Newnes (2014) 6. Barnard, M.E., Pyden, A., Rice, M.S., Linares, M., Tworoger, S.S., Howitt, B.E., Meserve, E.E., Hecht, J.L.: Inter-pathologist and pathology report agreement for ovarian tumor characteristics in the nurses’ health studies. Gynecologic oncology 150(3), 521–526 (2018) 7. Basile, V.: I… find
  4. Springer Nature Switzerland AG (2021) 8. Bechmann, A., Bowker, G.C.: Unsupervised by any other name: Hidden layers of knowledge production in artificial intelligence on social media. Big Data & Society 6(1), 2053951718819569 (2019) 9. Bender, E.M., Friedman, B.: Data statements for natural language… — [arXiv:2006.07159](https://arxiv.org/abs/2006.07159) find
  5. Cambridge University Press (2016) 12. Cabitza, F., Campagner, A.: The need to separate the wheat from the chaff in medical informatics. International Journal of Medical Informatics 152 (2021) 13. Cabitza, F., Campagner, A., Sconfienza, L.M.: As if sand were stone. new concepts and metrics to probe t… find
  6. Data Perspectivist turn 15. Campagner, A., Ciucci, D., Svensson, C.M., Figge, M.T., Cabitza, F.: Ground truthing from multi-rater labeling with three-way decision and possibility theory. Information Sciences 545, 771–790 (2021) 16. Chang, J.C., Amershi, S., Kamar, E.: Revolt: Collaborative crowdsour… find
  7. AAAI (2019) 26. Hildebrandt, M.: The issue of bias. the framing powers of ml. In: editor, T. (ed.) Machine Learning and Society: Impact, Trust, Transparency. find
  8. MIT Press (2021) 27. Jagabathula, S., Subramanian, L., Venkataraman, A.: Identifying unreliable and adversarial workers in crowdsourced labeling tasks. The Journal of Machine Learning Research 18(1), 3233–3299 (2017) 28. Jewett, M.A., Bombardier, C., Caron, D., Ryan, M.R., Gray, R.R., Louis, E.L.S.,… find
  9. V. Basile et al. 33. Linstone, H.A., Turoff, M.: The delphi method: Techniques and applications. In: The Delphi method: Techniques and applications. Addison-Wesley Publishing (1975) 34. Lok, C.E., Morgan, C.D., Ranganathan, N.: The accuracy and interobserver agreement in detecting the ‘gallop sounds… find
  10. Springer (2020) 41. Salminen, J.O., Al-Merekhi, H.A., Dey, P., Jansen, B.J.: Inter-rater agreement for social computing studies. In: 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS). pp. 80–87. find
  11. IEEE (2018) 42. Schaekermann, M., Goh, J., Larson, K., Law, E.: Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work. Proceedings of the ACM on Human-Computer Interaction 2(CSCW), 1–19 (2018) 43. Sharmanska, V., Hernández-Lobato, D., Miguel Hernandez-Lobato, J., Qu… find
  12. Springer (2019) 45. Svensson, C.M., Figge, M.T., Hübler, R.: Automated classification of circulating tumor cells and the impact of interobsever variability on classifier training and performance. Journal of Immunology Research 2015 (2015) 46. Tao, F., Jiang, L., Li, C.: Label similarity-based weigh… find
  13. Data Perspectivist turn 50. Yager, R.R.: Interval valued entropies for dempster–shafer structures. Knowledge-Based Systems 161, 390–397 (2018) 51. Yun, S., Oh, S.J., Heo, B., Han, D., Choe, J., Chun, S.: Re-labeling imagenet: from single to multi-labels, from global to localized labels. arXiv prepri… — [arXiv:2101.05022](https://arxiv.org/abs/2101.05022) find