Elamur Research⌕
☰
home / papers / 2026 / Stylistic Transfer from Annotator Communities to Large Language Models

Stylistic Transfer from Annotator Communities to Large Language Models

Paper · 2026 · Jay Chooi

Summary

This paper asks whether the paid annotators who supply human feedback during LLM post-training also pass their own variety of English on to the model. The author uses a natural experiment. Scale AI cut off its Remotasks platform in Nigeria between the releases of LLaMA 2 and LLaMA 3.1, so the first family was likely post-trained with Nigerian annotators and the second likely was not. Base and post-trained versions of both families answered prompts from AlpacaEval and MT-Bench. Their outputs were then compared with US English (COCA) and Nigerian English (ICE-Nigeria) using Jensen-Shannon distance over lowercased bigram distributions, combined in a difference-in-differences design. The two base models sit at about the same distance from both varieties. After post-training, LLaMA 2 moves toward Nigerian English, while LLaMA 3.1 moves away from it. A qualitative check shows that post-trained LLaMA 2 uses noticeably fewer contractions, which fits the formal register common in Nigerian English. One consequence the author raises is that AI-text detectors may wrongly flag human writers whose English resembles that of annotator communities.

Key points

  • The models compared were LLaMA-2-13b-hf and LLaMA-2-13b-chat-hf against LLaMA-3.1-8B and LLaMA-3.1-8B-Instruct. LLaMA 3 was left out because its post-training may have overlapped with the end of Remotasks access in Nigeria.
  • Generation used AlpacaEval and MT-Bench prompts, with a maximum of 512 new tokens, temperature 0.7 and top-p 0.9. It took about 2 hours on a g6e.xlarge AWS instance.
  • The reference corpora were an 8.9M-word sample of COCA for US English and the 400k-word written part of ICE-Nigeria for Nigerian English.
  • The core metric is ΔJSD, defined as JSD(model, US) minus JSD(model, Nigerian). The quantity of interest is the post-trained ΔJSD minus the base ΔJSD; a positive value means post-training brought the model closer to Nigerian English.
  • The author could not show a causal link between Nigerian annotators and the popularity of "delve", because such bigrams were too rare. Acknowledged limitations are the domain mismatch between the two reference corpora, other differences between the two post-training pipelines (such as SFT data), and the gap in model size (13B vs 8B).

Interesting ideas

  • I1. The design uses the base model as a built-in control. Measuring only the shift that post-training adds, and then comparing that shift across two families, removes much of the pre-training confound. This is a cheap way to attribute a change to a single pipeline stage.
  • I2. Features are ranked by multiplying two contrasts: corpus A vs corpus B, and treated model vs its control. The sign of one factor then indicates which explanation applies, which turns an aggregate distance into a short list of inspectable items.
  • I3. Jensen-Shannon distance on simple bigram distributions works as a bounded, symmetric, interpretable style metric that needs no trained classifier.
  • I4. The author deliberately skips the ambiguous middle case (LLaMA 3) so that the treated and untreated groups are clearly separated. Separating the groups this cleanly is a useful habit when the timing of an intervention is fuzzy.
  • I5. The paper explicitly flags domain mismatch between reference corpora as a rival explanation for any style difference. This is a reminder to match genres before claiming a variety effect.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when an LLM teacher tier or a new silver batch is introduced, compare the parser's output distribution before and after against a held-out reference. This would reveal whether the teacher is imprinting its own annotation habits rather than improving accuracy.
  • I2 (learn): rank dependency relations or lemma-relation pairs by "teacher vs gold" times "after vs before training" to find exactly which constructions silver data is pushing in a wrong direction.
  • I3 (back): compare generated text against human reference text with JSD over word or POS bigrams. This would catch stylistic drift in the text → graph → text compressor, such as missing contractions or an overly formal register.
  • I4 (learn): when evaluating whether a repair rule or data source helped, compare clearly before and clearly after checkpoints and exclude transitional ones, so the negative control stays clean.
  • I5 (tree): when converting UD dialects or mixing treebanks, check that apparent annotation-style differences are not just genre differences, for example by comparing on domain-matched subsets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harvard UniversityCambridge · United States112 works · 42,023 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jay Chooi. 2026. Stylistic Transfer from Annotator Communities to Large Language Models. Proceedings of the 10th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.latechclfl-1.13

@inproceedings{chooi2026stylistic,
  title = {{Stylistic Transfer from Annotator Communities to Large Language Models}},
  author = {Jay Chooi},
  booktitle = {Proceedings of the 10th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature 2026},
  year = {2026},
  doi = {10.18653/v1/2026.latechclfl-1.13},
  url = {https://doi.org/10.18653/v1/2026.latechclfl-1.13}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Russell Brandom. 2024. Scale AI’s Remotasks platform is dropping whole countries without explanation. find
  2. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, and 13 others. 2023. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv preprint. ArXiv:2307.15217 [cs]. ↗
  3. Ajewole-Orimogunje Christiana-Oluremi. 2013. Lexico-Grammatical Features of Nigerian English. International Journal of English Language Education, 1(1):pp. 261–271. ↗
  4. Mark Davies. 2008. The Corpus of Contemporary American English (COCA). find
  5. Johnson Ojeka Ebibi, Vera Anap Aaron, Songden Rejoice-James, and Ololade Adeyemo. 2018. A study on the use of contractions in speech and writing by selected students of the University of Jos. AFR-REV LALIGENS: An International Journal of Language, Literature and Gender Studies, 7(1):82. ↗
  6. Penelope Eckert and Sally McConnell-Ginet. 1992. Communities of Practice: Where Language, Gender, and Power All Live. In Kira Hall, Mary Bucholtz, and Birch Moonwomon, editors, Locating Power: Proceedings of the 1992 Berkeley Women and Language Conference, pages 89–99. Berkeley Women and Language Group, Berkeley. find
  7. Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and Linguistics Compass, 15(8):e12432. ↗
  8. Dirk Hovy and Shannon L. Spruit. 2016. The Social Impact of Natural Language Processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany. Association for Computational Linguistics. ↗
  9. ICE Project, University of Zürich & Chemnitz University of Technology / University of Bayreuth. 1999. ICE-East Africa (International Corpus of English): Kenya & Tanzania component. find
  10. ICE Project, University of Zürich & De La Salle University, Manila. 2004. ICE-Philippines (International Corpus of English). find
  11. Tom S. Juzek and Zina B. Ward. 2024. Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models. arXiv preprint. ArXiv:2412.11385 [cs] version: 1. ↗
  12. Tom S. Juzek and Zina B. Ward. 2025. Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback. ArXiv:2508.01930 [cs]. ↗
  13. Farooq Kperogi. 2010. Divided by a Common Language: A Comparison of Nigerian, American and British English. find
  14. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. Publication Title: GitHub repository. find
  15. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. 2023. GPT detectors are biased against non-native English writers. Patterns, 4(7):100779. ↗
  16. J. Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1):145–151. ↗
  17. Rosina Lippi-Green. 2012. English with an Accent: Language, Ideology and Discrimination in the United States, 2 edition. Routledge, London. find
  18. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. In Proceedings of the 40th International Conference on Machine Learning, pages 24950–24962. PMLR.
  19. Victoria University of Wellington, School of Linguistics & Applied Language Studies / University of Zürich, ICE Project. 1999. ICE-New Zealand (International Corpus of English). find
  20. Ding Wang, Shantanu Prabhat, and Nithya Sambasivan. 2022. Whose AI Dream? In search of the aspiration in data annotation. arXiv preprint. ArXiv:2203.10748 [cs]. ↗
  21. Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek Fai Wong, and Lidia Sam Chao. 2025. A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions. Computational Linguistics, 51(1):275–317. ↗
  22. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv preprint. ArXiv:2306.05685 [cs]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chooi-2026-stylistic-transfer-annotator-communities-large/