Elamur Research
☰
home / papers / 2023 / AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2303.16854

Summary

Manually labelling data for NLP is slow and expensive. This paper asks whether GPT-3.5 models can replace crowd workers. The authors build AnnoLLM, which gives the model the same materials a human annotator would get: a task description, label definitions and labelled examples. It then adds a two-stage "explain-then-annotate" procedure. First, the LLM sees each demonstration example with its gold label and writes a justification for that label. These self-written rationales are then used to build a few-shot chain-of-thought prompt for labelling new data. AnnoLLM was tested on three classification tasks: QK (relevance between a user query and keywords), BoolQ and WiC. It outperformed or matched crowdsourced annotators. The authors also used AnnoLLM as a filter to build ConIR, the first dataset for retrieving passages relevant to conversations. Retrievers trained only on MS-MARCO perform poorly on ConIR. The work matters for teams with small labelling budgets, since it shows that LLM labelling guided by explanations can approach human annotation quality.

Key points

  • On the QK test set (1,000 items), text-davinci-003 with a 4-shot CoT prompt (AnnoLLM) scored 75.60, compared with 71.5 for crowd workers, 70.00 for zero-shot and 67.80 for plain 8-shot prompting. Few-shot prompting without explanations was actually weaker than zero-shot.
  • Evaluation splits (dev/test):
    • QK: 350/1000
    • BoolQ: 3270/3245
    • WiC: 638/1400
  • ConIR was built in three steps, all using ChatGPT:
    • A source passage was first expanded so that the conversation would not simply copy its wording.
    • A multi-turn dialogue was then generated from the query and the expanded passage.
    • AnnoLLM discarded conversations that were not relevant to the original passage.
  • Retrieval results on ConIR (MRR@10):
    • DPR trained on MS-MARCO: 7.01
    • PROD trained on MS-MARCO: 10.61
    • DPR fine-tuned on ConIR: 19.32, with R@100 rising from 22.08 to 64.25
  • Human evaluation covered 100 conversations rated by three annotators:
    • fluency: 4.99/5
    • relevance: 2.53/3
    • factual consistency: 2.41/3
    • Fleiss' kappa: 0.55
  • The dataset and code are released at github.com/NLPCode/AnnoLLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

117 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 114 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.16854 · arXiv 2303.16854

@misc{he2023annollm,
  title = {{AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators}},
  author = {He, Xingwei and Lin, Zhenghao and Gong, Yeyun and Jin, A-Long and Zhang, Hang and Lin, Chen and Jiao, Jian and Yiu, Siu Ming and Duan, Nan and Chen, Weizhu},
  year = {2023},
  doi = {10.48550/arxiv.2303.16854},
  eprint = {2303.16854},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.16854}
}

References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. find our card
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. find our card
  4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. find
  5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL, pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  6. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  7. Hao Fu, Yao; Peng and Tushar Khot. 2022. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu’s Notion. find
  8. Xingwei He, A-Long Jin, Jun Ma, Yuan Yuan, and Siu Yiu. 2023. PivotFEC: Enhancing few-shot factual error correction with a pivot task approach using large language models. In Findings of EMNLP, pages 9960–9976, Singapore. Association for Computational Linguistics. find
  9. Xingwei He and Siu Ming Yiu. 2022. Controllable dictionary example generation: Generating example sentences for specific targeted audiences. In Proceedings of ACL, pages 610–627, Dublin, Ireland. Association for Computational Linguistics. find
  10. Xingwei He, Qianru Zhang, A-Long Jin, Jun Ma, Yuan Yuan, and Siu Ming Yiu. 2024. Improving factual error correction by learning to inject factual errors. In Proceedings of AAAI, volume 38, pages 18197– 18205. find
  11. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. find
  12. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745. find our card
  13. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP, pages 6769–6781, Online. Association for Computational Linguistics. find our card
  14. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  15. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of ACL, pages 6086–6096, Florence, Italy. Association for Computational Linguistics. find our card
  16. Zhenghao Lin, Yeyun Gong, Xiao Liu, Hang Zhang, Chen Lin, Anlei Dong, Jian Jiao, Jingwen Lu, Daxin Jiang, Rangan Majumder, et al. 2023. Prod: Progressive distillation for dense retrieval. In Proceedings of WWW, pages 3299–3308. find
  17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… find
  18. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of NAACL, pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  19. Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of ACL, pages 866–876, Melbourne, Australia. Association for Computational Linguistics. find our card
  20. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of NAACL, pages 5835–5847, Online. Association for Computationa… find our card
  21. Alec Radford. 2018. Improving language understanding by generative pre-training. OpenAI Technical Report. find
  22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. find
  23. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. find
  24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. find
  25. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
  26. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of ACL, pages 86–96, Berlin, Germany. Association for Computational Linguistics. find
  27. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631–1642, Seattle, Washington, USA. Association for Computational Ling… find
  28. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS, page 3104–3112, Cambridge, MA, USA. MIT Press. find
  29. James Thorne and Andreas Vlachos. 2021. Evidence-based factual error correction. In Proceedings of ACL, pages 3298–3309, Online. Association for Computational Linguistics. find
  30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, volume 30. Curran Associates, Inc. find
  32. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In NIPS, volume 32. Curran Associates, Inc. find our card
  33. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? GPT-3 can help. In Findings of EMNLP, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
  34. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. In ICLR. find our card
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NIPS, volume 35, pages 24824–24837. Curran Associates, Inc. find our card
  36. Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023. Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10205. find our card
  37. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In ICML, pages 11328–11339. PMLR. find our card
  38. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of ICCV, pages 2223–2232. find
  39. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906. find