📄 Statistical Methods for Annotation Analysis

Summary

This entry is a book review, not an original study. Rodrigo Wilkens assesses "Statistical Methods for Annotation Analysis" by Silviu Paun, Ron Artstein and Massimo Poesio (Springer, 2022). The book addresses a recurring problem in corpus building: when several coders label the same items, researchers must measure how reliably they agree and also derive one final label per item. It updates Artstein and Poesio's 2008 survey of agreement coefficients and adds topics that survey omitted, including latent models of agreement, probabilistic aggregation models and training directly on multi-coder data. Part 1 covers coefficients such as κ, π and α. It also covers design issues (missing labels, unit boundaries, skewed data) and how to interpret coefficient values. Part 2 presents models that infer labels from annotator behaviour, extending to a variational autoencoder, and methods for learning from disagreement with soft labels. The reviewer praises the book's critical and historical framing and its practical examples. He considers it valuable for researchers and for anyone validating an annotated dataset, although practitioners may find some task-specific sections less relevant.

Key points

• The book is volume 54 of Springer's Synthesis Lectures on Human Language Technologies series, edited by Graeme Hirst. It runs to xix+197 pages and is reviewed in Computational Linguistics 49(3), 2023.
• It builds on the 2008 article "Inter-coder Agreement for Computational Linguistics." It discusses Krippendorff's α, Scott's π and Cohen's κ, as well as weighted coefficients and the difficulty of interpreting them.
• Chapter 4 treats reliability as confidence estimation over latent item classes, introducing item difficulty. This moves beyond simple majority voting, which ignores coder bias.
• Chapter 5 covers probabilistic annotation models. These include hierarchical priors for annotators, inter-dependent labels (NER, information extraction), labels without predefined classes such as anaphora, and VAE-based aggregation.
• Chapter 6 covers learning from multi-annotated corpora through soft labels, individual coder labels and pooling of coder confusion. It closes with guidance on whether to aggregate annotations or keep them all, and on crowdsourced versus gold labels.