Elamur Research⌕
☰
home / papers / 2026 / Task Assignment meets Annotator Modeling: Human-LLM Collaborative Annotation with Constraints

Task Assignment meets Annotator Modeling: Human-LLM Collaborative Annotation with Constraints

Paper · 2026 · Kei Moriyama, Kouta Nakayama and Yukino Baba

Summary

Human crowdworkers and LLMs are both inexpensive sources of labels, but each is reliable on different kinds of items. Real projects also impose limits, such as caps on how many items one person may label or a total budget. Earlier work handled only one side. Annotator-modeling methods such as Learning to Defer and iCrowd ignore constraints. Constraint-aware assignment methods rely on historical accuracy and ignore task content.

The authors combine the two in a single pipeline:

  1. CrowdLayer is trained on past annotations. Its per-annotator confusion matrices, read as probabilities, predict how accurate each annotator will be on each new task, based on the task's features.
  2. These predictions become edge weights in a bipartite task–annotator graph.
  3. A linear program chooses the assignment that maximizes total expected accuracy while respecting the constraints.

For evaluation, they built and released TE-A, the TweetEval emoji task annotated by crowdworkers and an LLM. Across three scenarios and two constraint types, the method matched baseline accuracy without breaking any constraint. It was particularly robust when annotations were sparse. This makes mixed human–LLM labeling workable under real budgets.

Key points

  • Released resources: the TweetEval-Annotated (TE-A) dataset is on Hugging Face (keimoriyama/TweetEval_Annotated), and the code is on GitHub (babalablab/task-assignment).
  • Experimental setup: three scenarios were tested (large data, varying data size, varying annotation rate). Each was run under a maximum-assignment constraint and under a cost constraint.
  • Runtime test: a synthetic benchmark used 2D points in three classes and ten simulated annotators. The linear program took about 20 s for 10k tasks, about 79 s for 30k, and roughly 280–287 s for 100k. The authors consider this negligible next to crowdsourcing timelines measured in days.
  • Scope: the method is designed for classification. The authors propose reframing regression or generation work as binary validation tasks to extend it.
  • Limitations: experiments assign each task to exactly one annotator, so redundant labeling with majority voting is untested. Runtime also depends on the constraints and the hardware.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Science Applications · 102 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Kei Moriyama, Kouta Nakayama and Yukino Baba. 2026. Task Assignment meets Annotator Modeling: Human-LLM Collaborative Annotation with Constraints. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.79

@inproceedings{moriyama2026task,
  title = {{Task Assignment meets Annotator Modeling: Human-LLM Collaborative Annotation with Constraints}},
  author = {Kei Moriyama, Kouta Nakayama and Yukino Baba},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.79},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.79}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John P Dickerson, Karthik Abinav Sankararaman, Aravind Srinivasan, and Pan Xu. 2018. Assigning tasks to workers based on historical data: Online task assignment with two-sided arrivals. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AA-MAS 2018, page… find
  2. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650.
  3. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, pages 11173–11195.
  4. Shahzad Sarwar Bhatti, Jiahao Fan, Kangrui Wang, Xiaofeng Gao, Fan Wu, and Guihai Chen. 2021. An approximation algorithm for bounded task assignment problem in spatial crowdsourcing. IEEE Transactions on Mobile Computing, 20(8):2536–2549. find
  5. Ju Fan, Guoliang Li, Beng Chin Ooi, Kian-Lee Tan, and Jianhua Feng. 2015. iCrowd: An adaptive crowdsourcing framework. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD 2015, pages 1015–1030. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… find
  7. Virginia K Felkner, Jennifer A Thompson, and Jonathan May. 2024. GPT is not an annotator: The necessity of human annotation in fairness benchmark construction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, pages 14104– 1… find
  8. Chengzhi Cao, Yinghao Fu, Sheng Xu, Ruimao Zhang, and Shuang Li. 2024. Enhancing human-AI collaboration through logic-guided reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024. find
  9. X Chen, Qihang Lin, and Dengyong Zhou. 2013. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In International Conference on Machine Learning, volume 28 of ICML 2013, pages 64–72. find
  10. Zhendong Chu, Jing Ma, and Hongning Wang. 2021. Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35 of AAAI 2021, pages 5832–5840. find
  11. Patrick Hemmer, Sebastian Schellhammer, Michael Vössing, Johannes Jakubik, and Gerhard Satzger. 2022. Forming effective human-AI teams: Building machine learning models that complement the capabilities of multiple experts. In Proceedings of the Thirty-First International Joint Conference on Artifici… find
  12. Chien-Ju Ho, S Jabbari, and Jennifer Wortman Vaughan. 2013. Adaptive task assignment for crowdsourced classification. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of ICML 2013, pages 534–542. find
  13. Chien-Ju Ho and Jennifer Vaughan. 2021. Online task assignment in crowdsourcing markets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26 of AAAI 2012, pages 45–51. find
  14. Maliha Jahan, Helin Wang, Thomas Thebaud, Yinglun Sun, Giang Ha Le, Zsuzsanna Fagyal, Odette Scharenborg, Mark Hasegawa-Johnson, Laureano Moro Velazquez, and Najim Dehak. 2024. Finding spoken identifications: Using GPT-4 annotation for an efficient and fast dataset creation pipeline. In Proceedings… find
  15. Manas Joglekar, Hector Garcia-Molina, and Aditya Parameswaran. 2015. Comprehensive and reliable crowd assessment algorithms. In 2015 IEEE 31st International Conference on Data Engineering, ICDE 2015, pages 195–206. find
  16. Asif R Khan and Hector Garcia-Molina. 2017. Crowd-DQS: Dynamic question selection in crowdsourcing systems. In Proceedings of the 2017 ACM International Conference on Management of Data, SIG-MOD 2017, pages 1447–1462. find
  17. Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. 2024. MEGAnno+: A human-LLM collaborative annotation system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, EACL 2024, pages 168–176. find
  18. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations, ICLR 2015.
  19. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. WANLI: Worker and AI collaboration for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847.
  20. David Madras, Toniann Pitassi, and Richard Zemel. 2017. Predict responsibly: Improving fairness and accuracy by learning to defer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NeurIPS 2017, pages 6150–6160. find
  21. Xin Miao, Yanrong Kang, Qiang Ma, Kebin Liu, and Lei Chen. 2020. Quality-aware online task assignment in mobile crowdsourcing. ACM transactions on sensor networks, 16(3):1–21. find
  22. Luyi Mo, Reynold Cheng, Ben Kao, Xuan S Yang, Chenghui Ren, Siyu Lei, David W Cheung, and Eric Lo. 2013. Optimizing plurality for human intelligence tasks. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, CIKM 2013, pages 1929– 1938. find
  23. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17.
  24. Hussein Mozannar and David Sontag. 2020. Consistent estimators for learning to defer to an expert. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, pages 7076–7087. find
  25. Harikrishna Narasimhan, Wittawat Jitkrittum, A Menon, A Rawat, and Surinder Kumar. 2022. Post-hoc estimators for learning to defer to an expert. In Advances in Neural Information Processing Systems 35, NeurIPS 2022, pages 29292–29304. find
  26. Maja Pavlovic and Massimo Poesio. 2024. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP, LREC-COLING 2024, pages 100–110.
  27. Chenxi Qiu, Anna C Squicciarini, Barbara Carminati, James Caverlee, and Dev Rishi Khare. 2016. Crowd-Select: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management,… find
  28. Charvi Rastogi, Marco Tulio Ribeiro, Nicholas King, Harsha Nori, and Saleema Amershi. 2023. Supporting human-AI collaboration in auditing LLMs with LLMs. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2023, pages 913–926. find
  29. Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, 11(43):1297– 1322. find
  30. Filipe Rodrigues and Francisco Pereira. 2017. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI 2017, pages 1611–1618. find
  31. Ashish Sharma, Sudha Rao, Chris Brockett, Akanksha Malhotra, Nebojsa Jojic, and William B Dolan. 2024. Investigating agency of LLMs in human-AI collaboration tasks. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  32. Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silberman. 2019. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, CVPR 2019, pages 11244– 11253. find
  33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moriyama-2026-task-assignment-meets-annotator-modeling/