Elamur Research⌕
☰
home / papers / 2026 / SALOMO: An Annotation Tool for Complex Annotation Tasks with a Large Number of Labels

SALOMO: An Annotation Tool for Complex Annotation Tasks with a Large Number of Labels

Summary

Assigning labels from a very large inventory overwhelms human annotators. The number of options exceeds what short-term memory can handle, which makes the work slow and inconsistent. In a pilot, students coding news sentences against an earlier 31-type media bias taxonomy needed about three minutes per sentence, found it very hard, and agreed poorly. SALOMO (Smart Annotation Labeling On Model Outputs) responds by letting two LLMs, GPT-5 and GPT-o3, pre-annotate each item with the same prompt. Labels both models choose count as accepted by default. Humans mainly act as tiebreakers where the models disagree, though they can still fix obvious errors. The tool is a single local HTML5 page with category guidelines shown on hover. The case study uses a new fine-grained news bias taxonomy (37 types; 38 in the final version). In a within-subject user study with 21 participants, pre-selection made annotation faster and lowered perceived difficulty. Participants also gained confidence and applied more varied labels. However, they leaned strongly toward the suggested labels, so the quality of the LLM pre-selection becomes critical. The authors release a small annotated dataset of biased sentences.

Key points

  • The two LLMs label at different rates: GPT-o3 averages about 1.68 labels per sentence and GPT-5 about 3.0. Their agreement is moderate (Jaccard 0.503, micro-averaged Cohen's κ = 0.581, bootstrap-adjusted F1 = 0.514 over 500 samples). Smaller and open-source models performed clearly worse and were not used.
  • In the pilot, annotators took 90 seconds to seven minutes per sentence on 25 sentences.
  • In the 21-person study, the pre-selection condition was significantly faster, with 33.5 s reported as the average. Participants assigned more and more diverse labels, rated the task easier, and felt more confident.
  • Pre-selected labels proved "sticky", nudging annotators toward stylistic and subjective bias categories. Participants also admitted getting sloppier on later sentences.
  • Pre-annotation currently runs as an external Python script. The tool will be released on GitHub, and future work may add server-based team collaboration.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4odk4p62uz9p

@inproceedings{menzner2026salomo,
  title = {{SALOMO: An Annotation Tool for Complex Annotation Tasks with a Large Number of Labels}},
  author = {Menzner, Tim},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4odk4p62uz9p},
  url = {https://doi.org/10.63317/4odk4p62uz9p}
}

References

27 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Beck, Stephanie Eckman, Bolei Ma, Rob Chew, and Frauke Kreuter. 2024. Order effects in annotation tasks: Further evidence of annotation sensitivity. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pages 81–86, St Julians, Malta. Association for Computational Lin… find
  2. Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele, Grigori Sidorov, Dietrich Klakow, Philip Slusallek, Olga Kolesnikova, and Seid Muhie Yimam. 2025. Evaluating the capabilities of large language models for multi-label emotion understanding. In Proceedings of the 31st International Conferenc… find
  3. Max Bernhardt, Daniel C. Castro, Ryutaro Tanno, et al. 2022. Active label cleaning for improved dataset quality under resource constraints. Nature Communications, 13(1):1161. find
  4. Nelson Cowan. 2001. The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24(1):87–114. find
  5. Jia Deng, Olga Russakovsky, Jonathan Krause, Michael S. Bernstein, Alex Berg, and Li Fei-Fei. 2014. Scalable multi-label annotation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, page 3099–3102, New York, NY, USA. Association for Computing Machinery. find
  6. Bryan Guan, Tanya G. Roosta, Peyman Passban, and Mehdi Rezagholizadeh. 2025. The order effect: Investigating prompt sensitivity in closed-source llms. arXiv preprint arXiv:2502.04134. Accessed: 2025-09-10. — [arXiv:2502.04134](https://arxiv.org/abs/2502.04134) find
  7. Austin Z. Henley and David Piorkowski. 2024. Supporting annotators with affordances for efficiently labeling conversational data. arXiv preprint arXiv:2403.07762. — [arXiv:2403.07762](https://arxiv.org/abs/2403.07762) find
  8. HumanSignal. 2025. Label studio: Open source data labeling. Accessed: 2025-09-10. find
  9. Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang. 2024. MEGAnno+: A human-LLM collaborative annotation system. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 168–176, St. Julians, Ma… find
  10. J. C. Klie, R. E. de Castilho, T. Neumann, S. G. Günter, and M. Gertz. 2018. The inception platform: Machine-assisted and knowledge-driven web annotation. In Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018), pages 1–10. Association for Computational Linguis… find
  11. Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology, 4th edition. SAGE Publications, Thousand Oaks, CA. find
  12. Johann Laux, Fabian Stephany, and Alice Lief-green. 2025. Better pay, clearer guidance: Investing in the working conditions of artificial intelligence data workers. Big Data & Society, 12(2):20539517251351320. find
  13. Jochen L. Leidner and Luca Jung. 2024. TAME II: A modern geographic text annotation tool. In Proceedings of the 21st International Symposium on Web and Wireless Geographical Information Systems (W2GIS 2024), volume 14673 of Lecture Notes in Computer Science, pages 95–104. Springer Nature. find
  14. Anna Sofia Lippolis, Mohammad Javad Saeedizade, Robin Keskisärkkä, Aldo Gangemi, Eva Blomqvist, and Andrea Giovanni Nuzzolese. 2025. Large language models assisting ontology evaluation. arXiv preprint arXiv:2507.14552. — [arXiv:2507.14552](https://arxiv.org/abs/2507.14552) find
  15. Marcus Ma, Georgios Chochlakis, Niyantha Maruthu Pandiyan, Jesse Thomason, and Shrikanth Narayanan. 2025. Large language models do multi-label classification differently. arXiv preprint arXiv:2505.17510. find
  16. Adrien Marchal, Merel C. J. Scholman, Frances Yung, and Vera Demberg. 2022. Establishing annotation quality in multi-label annotations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3658–3671, Gyeongju, Republic of Korea. International Committee on Computati… find
  17. Winter Mason and Duncan J. Watts. 2010. Financial incentives and the "performance of crowds". SIGKDD Explor. Newsl., 11(2):100–108. find
  18. Tim Menzner and Jochen L. Leidner. 2026. The table of media bias elements: A sentence-level taxonomy of media bias types and propaganda techniques. arXiv preprint arXiv:2601.05358. — [arXiv:2601.05358](https://arxiv.org/abs/2601.05358) find
  19. George A. Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2):81–97. find
  20. Minxue Niu, Mimansa Jaiswal, and Emily Mower Provost. 2024. From text to emotion: Unveiling the emotion annotation capabilities of llms. In Interspeech 2024. find
  21. Maja Pavlovic and Massimo Poesio. 2024. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024, pages 100–110, Torino, Italia. ELRA and… find
  22. Hope Schroeder, Deb Roy, and Jad Kabbara. 2025. Just put a human in the loop? investigating llm-assisted annotation for subjective tasks. In Findings of the Association for Computational Linguistics: ACL 2025, pages 25771–25795, Vienna, Austria. Association for Computational Linguistics. find
  23. Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Martin Santillan Cooper, Dina Epelboim, Zheng Zhang, Dakuo Wang, Lucy Yip, Liat Ein-Dor, Lena Dankin, Ilya Shnayderman, Ranit Aharonov, Yunyao Li, Naftali Liberman, Philip Levin Slesarev, Gwilym Newton, Shila Of… find
  24. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin… find
  25. Petter Törnberg. to appear. Large language models outperform expert coders and supervised classifiers at annotating political social media messages. Social Science Computer Review, page 08944393241286471. find
  26. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
  27. Yifan Wang, David Stevens, Pranay Shah, Wenwen Jiang, Miao Liu, Xu Chen, Robert Kuo, Na Li, Boying Gong, Daniel Lee, Jiabo Hu, Ning Zhang, and Bob Kamma. 2024. Model-in-the-loop (milo): Accelerating multimodal ai data annotation with llms. arXiv preprint arXiv:2409.10702. Accessed: 2025-09-10. — [arXiv:2409.10702](https://arxiv.org/abs/2409.10702) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/menzner-2026-salomo-annotation-tool-complex-annotation/