Elamur Research⌕
☰
home / papers / 2026 / Disentangling Annotator Skill from Verifier Strictness in Cross-Verified Dialogue Annotation

Disentangling Annotator Skill from Verifier Strictness in Cross-Verified Dialogue Annotation

Summary

When a second person reviews and corrects an annotated file, the number of corrections reflects two things at once: how accurate the annotator was and how demanding the reviewer is. This paper separates those two factors using verification records from RASwDA, a manual re-alignment in Praat of Switchboard Dialog Act (SwDA) intervals to the original audio. Five team members produced the records in 2024–2025. The review schedule was crossed: each of three annotators had work checked by four different verifiers. This design allowed the authors to fit a cross-classified mixed-effects model on corrections per non-silence interval. The model includes annotator, verifier, batch and file-level components.

Annotator identity accounts for a modest share of the variance. Verifier identity accounts for none, and this held across several sensitivity refits. Most corrections were audio-anchored timestamp fixes rather than label judgments, which likely leaves little room for verifier strictness to matter. The authors present this as a corpus-specific null result with limited statistical power. The main transferable contribution is the method, supported by the released action-typed logs.

Key points

  • Released resource: 977 machine-readable *_changes.csv logs covering 1,045 Switchboard file sides and 105,005 corrections. Each correction is tagged with one of five action types (BOUNDARY_SHIFT, INSERT, DELETE, DA_CHANGE, MODIFY), along with time intervals and DA labels before and after the fix.
  • The model was fit on 537 complete-case file sides and split the variance as follows:
    • annotator: 10.8%
    • batch-within-annotator: 8.7%
    • file-level residual: 80.6%
    • verifier: 0 (singular fit; this result persisted in all seven refits, which either dropped one unit at a time or used a different response variable)
  • Boundary placement makes up 58.5% of all corrections and is the dominant type for every annotator. DA-label changes are only 3.0%.
  • 503 file sides have no recoverable annotator. These files differ statistically from the attributed ones (longer, with fewer corrections), so the findings apply only to the complete-case subset.
  • With only three annotators and five verifiers, the zero verifier effect means "no detectable effect" rather than proof that verifiers are interchangeable. Tasks dominated by subjective labeling might show real verifier strictness effects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigdial-1.29

@inproceedings{tao2026disentangling,
  title = {{Disentangling Annotator Skill from Verifier Strictness in Cross-Verified Dialogue Annotation}},
  author = {Tao, Zihao and Prado, John A. and LaManna, Ignazio Steven and Puterbaugh, Ryan and Datta, Mim and Hirschberg, Julia},
  booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2026},
  doi = {10.18653/v1/2026.sigdial-1.29},
  url = {https://doi.org/10.18653/v1/2026.sigdial-1.29}
}

References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James Allen and Mark Core. 1997. DAMSL: Dialog act markup in several layers. Draft, University of Rochester. find
  2. R. H. Baayen, D. J. Davidson, and D. M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412. find
  3. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1– 48. find our card
  4. Petra Saskia Bayerl and Karsten Ingmar Paul. 2011. What determines inter-coder agreement in manual annotations? A meta-analytic investigation. Computational Linguistics, 37(4):699–725. find
  5. Paul Boersma and David Weenink. 2024. Praat: Doing phonetics by computer. Computer program. Version 6.4, retrieved 2024. find
  6. A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20–28. find
  7. Thomas Eckes. 2019. Many-Facet Rasch Measurement, pages 153–175. Routledge. find
  8. John J. Godfrey, Edward C. Holliman, and Jane Mc-Daniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), volume 1, pages 517–520, San Francisco, CA. IEEE. find
  9. Daniel Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado, Boulder. find
  10. Omer Nahum, Nitay Calderon, Orgad Keller, Idan Szpektor, and Roi Reichart. 2025. Are LLMs better than reported? Detecting label errors and mitigating their effect on model performance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26782–26809, Suzho… find
  11. No annotator-by-verifier interaction term. Equation 1 treats annotator and verifier effects as 421 find
  12. Shinichi Nakagawa and Holger Schielzeth. 2013. A general and simple method for obtaining R 2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2):133–142. find
  13. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311– 326. find our card
  14. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing Bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571–585. find
  15. Elizabeth Shriberg, Rebecca Bates, Andreas Stolcke, Paul Taylor, Daniel Jurafsky, Klaus Ries, Noah Coccaro, Rachel Martin, Marie Meteer, and Carol Van Ess-Dykema. 1998. Can prosody aid the automatic classification of dialog acts in conversational speech? Language and Speech, 41(3–4):443–492. find
  16. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339–374. find our card
  17. Michiel van der Meer, Neele Falk, Pradeep K. Murukannaiah, and Enrico Liscio. 2024. Annotator-centric active learning for subjective NLP tasks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18537–18555, Miami, Florida, USA. Association for Computati… find
  18. Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christopher Homan. 2023. Disagreement matters: Preserving label diversity by jointly modeling item and annotator label distributions with DisCo. In Findings of the Association for Computational Linguistics: ACL… find
  19. Wenbo Zhang and Yuhan Wang. 2025. Act2P: LLM-driven online dialogue act classification for power analysis. In Findings of the Association for Computational Linguistics: ACL 2025, pages 20494–20504, Vienna, Austria. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tao-2026-disentangling-annotator-skill-verifier-strictness/