Elamur Research⌕
☰
home / papers / 2026 / Transcription Accuracy in the Icelandic Gigaword Corpus: Evaluating Automatic and Manual Annotation

Transcription Accuracy in the Icelandic Gigaword Corpus: Evaluating Automatic and Manual Annotation

Summary

Sociolinguists increasingly rely on large automatically annotated corpora. But transcripts may not faithfully reflect what speakers actually said, and this source of error is rarely examined. The authors study this for Stylistic Fronting (SF), an optional word-order pattern that signals formal style in Icelandic, using the Icelandic Parliament Corpus within the Icelandic Gigaword Corpus.

They compare two sources of SF data. One is automatic extraction from official transcripts written by parliamentary staff, tagged with ABLTagger 3.0.0, lemmatized with Nefnir, and filtered by a Python script. The other is manual correction made by listening to the original audio recordings, covering 2005–2021 for a balanced sample of 20 MPs.

Staff transcribers frequently insert SF that the speaker never produced, so the automatic data show somewhat higher SF rates than the audio-based coding. Even so, both datasets trace the same changes in SF over 16 years. Transcript-based automatic coding therefore remains a usable basis for studying language change across a speaker's lifespan, provided that its upward bias in SF rates is taken into account.

Key points

  • Data: 37,219 automatically extracted SF-eligible tokens from 20 MPs, of which 25,096 (67.43%) were manually checked against audio. The MPs were balanced by gender (10/10), birth year (1942–1990) and eight political parties.
  • Reliability of manual coding: A doubly coded subsample of 980 tokens reached 99.29% inter-annotator agreement (Cohen's Kappa = 0.96).
  • Transcript–audio mismatches:
    • Transcribers added SF in 2,627 cases.
    • They removed SF in only 142 cases.
    • In 812 cases, rewording created an SF context absent from the speech; these tokens were excluded.
  • Source of bias: The automatic tools themselves showed no errors on this construction (relative clauses with sem). The discrepancy comes from editorial normalization of transcripts toward standard written Icelandic.
  • ASR not used: Earlier work found ASR quality on this corpus too low to avoid manual correction. Audio is only authentic from 2005 onward, since earlier recordings are synthesized from the transcripts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4f2rpzig5h8p

@inproceedings{mechler2026transcription,
  title = {{Transcription Accuracy in the Icelandic Gigaword Corpus: Evaluating Automatic and Manual Annotation}},
  author = {Mechler, Johanna and Stefánsdóttir, Lilja Björk and Ingason, Anton Karl},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4f2rpzig5h8p},
  url = {https://doi.org/10.63317/4f2rpzig5h8p}
}

References

34 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ásgrímur Angantýsson. 2017. Stylistic Fronting and related constructions in the Insular Scandinavian languages. In Höskuldur Þráinsson, Caroline Heycock, and Zakaris Svabo, editors, Syntactic Variation in Scandinavian. Studies in Germanic Linguistics, pages 277–306. John Benjamins Publishing Company, Netherlands. ↗
  2. Starkaður Barkarson, Steinþór Steingrímsson, and Hildur Hafsteinsdóttir. 2022. Evolving large text corpora: Four versions of the Icelandic gigaword corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2371–2381. find
  3. Mary Bucholtz. 2000. The politics of transcription. Journal of Pragmatics, 32(10):1439–1465. ↗
  4. Mary Bucholtz. 2007. Variation in transcription. Discourse Studies, 9(6):784–808. ↗
  5. Alessandro Duranti. 1997. Linguistic Anthropology. Cambridge University Press, Cambridge. find
  6. Janet A. Edwards and Martin D. Lampert. 1993. Talking data: Transcription and coding in discourse research. Lawrence Erlbaum, Hillsdale, New Jersey. find
  7. Judy Y Fong, Michal Borsky, Inga R Helgadóttir, and Jon Gudnason. 2018. Manual post-editing of automatically transcribed speeches from the Icelandic parliament - Althingi. arXiv preprint arXiv:1807.11893. ↗
  8. Andie Sophia Fontaine. 2015. Iceland’s finance minister amongst Ashley Madison users. Reykjavík Grapevine. https://grapevine.is/news/2015/08/31/icelands-finance-minister-amongst-ashley-madison-users/. ↗
  9. Eugenie Giesbrecht and Stephanie Evert. 2009. Is part-of-speech tagging a solved task? An evaluation of POS taggers for the German web as corpus. In Proceedings of the Fifth Web as Corpus Workshop, pages 27–35. find
  10. Anders Holmberg. 2000. Scandinavian Stylistic Fronting: How any category can become an expletive. Linguistic Inquiry, 31(3):445–483. ↗
  11. Anders Holmberg. 2006. Stylistic Fronting. The Blackwell Companion to Syntax, pages 532– 565. find
  12. Nancy Ide and Chris Brew. 2000. Requirements, tools, and architectures for annotated corpora. In Proceedings of data architectures and software support for large corpora, pages 1–5. find
  13. Anton Karl Ingason, Sigrún Helgadóttir, Hrafn Loftsson, and Eiríkur Rögnvaldsson. 2008. A mixed method lemmatization algorithm using a Hierarchy of Linguistic Identities (HOLI). In Advances in Natural Language Processing, pages 205–216, Berlin, Heidelberg. Springer. find
  14. Anton Karl Ingason and Jim Wood. 2017. Clause bounded movement: Stylistic Fronting and phase theory. Linguistic Inquiry, 48(3):513–527. ↗
  15. Svanhvít Lilja Ingólfsdóttir, Hrafn Loftsson, Jón Friðrik Daðason, and Kristín Bjarnadóttir. 2019. Nefnir: A high accuracy lemmatizer for Icelandic. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages 310–315. ↗
  16. Hrafn Loftsson. 2008. Tagging Icelandic text: A linguistic rule-based approach. Nordic Journal of Linguistics, 31(1):47–72. ↗
  17. Brian MacWhinney. 2000. The CHILDES Project: The Database, volume 2. Psychology Press, New York. find
  18. Brian MacWhinney. 2014. The CHILDES Project: Tools for Analyzing Talk, Volume I: Transcription Format and Programs. Psychology Press, New York. find
  19. Joan Maling. 1980. Inversion in embedded clauses in Modern Icelandic. Íslenskt mál, 2:175–193. find
  20. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313– 330. ↗ our card
  21. Johanna Mechler, Lilja Björk Stefánsdóttir, and Anton Ingason. 2025. Language use of political parties over time: Stylistic fronting in the icelandic gigaword corpus. In Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities, pages 313–318. find
  22. Elinor Ochs. 1979. Transcription as theory. In E. Ochs and B. B. Schieffelin, editors, Developmental pragmatics, pages 43–72. Academic Press, New York. find
  23. R Core Team. 2024. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. find
  24. David Sankoff and Suzanne Laberge. 1978. The linguistic market and the statistical explanation of variability. In David Sankoff, editor, Linguistic Variation: Models and Methods, pages 239–250. Academic Press, New York. find
  25. Elizabeth Shriberg, Andreas Stolcke, and Don Baron. 2001. Observations on overlap: findings and implications for automatic processing of multi-party conversation. In Interspeech, pages 1359–1362. find
  26. Skrifstofa Alþingis. 2017. Háttvirtur þingmaður – handbók um þingstörfin. Published on the website of the Icelandic parliament. find
  27. Lilja Björk Stefánsdóttir and Anton Karl Ingason. 2018. A high definition study of syntactic lifespan change. U. Penn Working Papers in Linguistics, 24(1):1–10. find
  28. Lilja Björk Stefánsdóttir and Anton Karl Ingason. 2024a. Reinventing an identity for a more liberal audience. New Ways of Analyzing Variation, 52. find
  29. Lilja Björk Stefánsdóttir and Anton Karl Ingason. 2024b. Using the Icelandic Gigaword Corpus to explain lifespan change. In Proceedings of CLARIN Annual Conference 2024, pages 6–9. find
  30. Lilja Björk Stefánsdóttir and Anton Karl Ingason. 2024c. Wiggly lifespan change in a crisis. contrasting reactive and proactive identity construction. U. Penn Working Papers in Linguistics, 30(2):119–125. Lilja Björk Stefánsdóttir, Johanna Mechler, and Anton Karl Ingason. Forthcoming. The stylistic aftermath of the financial crisis in the Icelandic Parliament: Reinventing identity at the level of political parties. U. Penn Working Papers in Linguistics. find
  31. Lilja Björk Stefánsdóttir and Anton Karl Ingason. 2025. Transcriber effects in the Icelandic parliament corpus. Digital Humanities in the Nordic and Baltic Countries Publications, 7(1):106– 110. ↗
  32. Steinþór Steingrímsson, Sigrún Helgadóttir, Eiríkur Rögnvaldsson, Starkaður Barkarson, and Jón Guðnason. 2018. Risamálheild: A very large Icelandic text corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, pages 4361–4366, Miyazaki, Japan. find
  33. Steinþór Steingrímsson, Örvar Kárason, and Hrafn Loftsson. 2019. Augmenting a BiLSTM tagger with a morphological lexicon and a lexical category identification step. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1161–1168, Varna, Bulgaria. find
  34. Höskuldur Thráinsson. 2007. The Syntax of Icelandic. Cambridge University Press, Cambridge. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mechler-2026-transcription-accuracy-icelandic-gigaword-corpus/