Elamur Research⌕
☰
home / papers / 2018 / Comparing Bayesian Models of Annotation

Comparing Bayesian Models of Annotation

Paper · 2018 · Silviu Paun, Bob Carpenter, Jon Chamberlain et al.

Summary

In NLP, crowdsourced labels are usually merged by majority vote and checked with chance-corrected agreement coefficients. Both approaches treat annotators as equally competent and ignore bias and item effects. Probabilistic annotation models address these issues, but they had rarely been compared directly on real data. Paun et al. benchmark six Bayesian models that differ in how parameters are shared:

  • a fully pooled multinomial model;
  • unpooled models (D&S, MACE);
  • partially pooled hierarchical models (HIER D&S, ITEMDIFF, LOGRNDEFF).

Several of these were generalized from binary to multiclass labels. All are implemented in Stan with full posterior inference. They are judged on four criteria: match with gold labels, prediction of new annotations, annotator characterization, and item difficulty. The test data are four datasets with different shares of spam-like annotators.

Models that represent annotators explicitly generally perform best, though unpooled ones can overfit. HIER D&S is usually strongest, except on a game-collected dataset where annotators fall into two quality groups. The study gives practical guidance on choosing and implementing annotation models instead of defaulting to majority voting.

Key points

  • The six models compared are MULTINOM (pooled), D&S and MACE (unpooled), and HIER D&S, ITEMDIFF and LOGRNDEFF (partially pooled via hierarchical priors).
  • All models were written in Stan using Hamiltonian Monte Carlo. A non-centered parameterization proved essential for fitting the hierarchical models reliably.
  • Label switching was handled by relabeling each chain after inference. The relabeling used a small gold sample of about five items per class to pick the permutation with the highest accuracy.
  • HIER D&S usually leads on the Snow et al. (2008) Amazon Mechanical Turk datasets. On the game-with-a-purpose PD dataset, MACE and D&S beat it, because a few strong annotators coexist with many weak ones there.
  • The authors suggest that a multi-modal hierarchical prior might suit two-population data like PD better. Supplementary material is published by the DALI project at QMUL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States149 works · 48,852 citations🎓University of EssexColchester · United Kingdom25 works · 3,285 citations🎓Queen Mary University of LondonLondon · United Kingdom39 works · 3,307 citations🎓Bocconi UniversityMilan · Italy16 works · 2,850 citations

📚 Shelves 8 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Social Sciences · 744 works Decision Sciences · 85 works General Social Sciences · 87 works Statistics, Probability and Uncertainty · 32 works

Cite

Cite as Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz and Massimo Poesio. 2018. Comparing Bayesian Models of Annotation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00040

@article{paun2018comparing,
  title = {{Comparing Bayesian Models of Annotation}},
  author = {Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz and Massimo Poesio},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00040},
  url = {https://doi.org/10.1162/tacl_a_00040}
}

References

5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paul S. Albert and Lori E. Dodd. 2004. A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. Biometrics, 60(2):427–435. find
  2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. find
  3. James O. Berger. 2013. Statistical Decision Theory and Bayesian Analysis. Springer. find
  4. José M. Bernardo and Adrian F. M. Smith. 2001. Bayesian Theory. IOP Publishing. find
  5. Michael Betancourt and Mark Girolami. 2015. Hamiltonian Monte Carlo for hierarchical models. Current trends in Bayesian methodology with applications, 79:30. \#\# Conclusions find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paun-2018-comparing-bayesian-models-annotation/