Elamur Research⌕
☰
home / papers / 2023 / Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost

Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost

Paper · 2023 · Parikshit Bansal and Amit Sharma

Summary

Supervised NLP models perform well on familiar data but often make errors on new domains or rare subgroups for which few labeled examples are available. The authors, from Microsoft Research India, propose labeling unlabeled data with an LLM and then fine-tuning a compact model on these labels. Since the number of LLM queries is limited by budget, the key question is which examples should be sent for labeling. Random selection yields little benefit, and classical active learning based on model uncertainty also performs poorly. The authors therefore introduce the Conditional Informativeness metric. It measures how much the score of the fine-tuned model differs from the score of the base model (e.g., BERT) from which it was derived. The largest disagreement points to likely errors, so these are the examples sent to the LLM. The method was evaluated on semantic similarity (Quora) and semantic search (Wikipedia, Amazon). It improved accuracy on both the target and source domains. In practice, this offers an inexpensive way to adapt a model to a new domain without manual labeling.

Key points

  • Data: semantic similarity (duplicate question detection) on Quora. Semantic search on LF-WikiSeeAlsoTitles-320K (target domain "USA") and LF-AmazonTitles-131K (target domains "Books" and "Kitchen and Dining"). The metric for search is Precision@1.
  • Difference between LLM labels and ground-truth (GT) labels: with LLM labels, uncertainty-based selection loses to Conditional Informativeness. With GT labels, the picture is reversed: uncertainty works better. The authors explain this by the fact that the LLM makes fewer errors than the base model precisely where the disagreement is large.
  • Semantic search: the best overall accuracy and target-domain accuracy were obtained by LLM labeling of the 40% of examples with the highest Conditional Informativeness.
  • P@1 figures (base model → fine-tuned → fine-tuned with LLM labels): Wikipedia (USA) — 12.33 → 12.54 → 12.95; Amazon (Books) — 16.78 → 17.37 → 18.28.
  • Limitations: for the Kitchen domain, LLM labeling yielded no gain (32.91, the same as the base model). There, product similarity is well captured by lexical overlap, whereas books require world knowledge, such as a shared author. For such domains, the authors recommend regularization toward the base model instead of an LLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft ResearchBengaluru · India19 works · 4,213 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Parikshit Bansal and Amit Sharma. 2023. Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost. arXiv:2306.15766.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.15766 · arXiv 2306.15766

@misc{bansal2023large,
  title = {{Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost}},
  author = {Parikshit Bansal and Amit Sharma},
  year = {2023},
  doi = {10.48550/arxiv.2306.15766},
  eprint = {2306.15766},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.15766}
}

References

36 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Bhatia, K. Dahiya, H. Jain, P. Kar, A. Mittal, Y. Prabhu, and M. Varma. 2016. The extreme classification repository: Multi-label datasets and code. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.
  4. Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2023. An empirical survey of data augmentation for limited data learning in nlp. Transactions of the Association for Computational Linguistics, 11:191–211.
  5. Qinyuan Cheng, Xiaogui Yang, Tianxiang Sun, Linyang Li, and Xipeng Qiu. 2023. Improving contrastive learning of sentence embeddings from ai feedback. arXiv preprint arXiv:2305.01918. find
  6. David D Lewis. 1995. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum, volume 29, pages 13–19. ACM New York, NY, USA. find
  7. Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. 2023. Chatgpt goes to law school. Available at SSRN. find
  8. Bo Li, Yezhen Wang, Tong Che, Shanghang Zhang, Sicheng Zhao, Pengfei Xu, Wei Zhou, Yoshua Bengio, and Kurt Keutzer. 2020. Rethinking distributional matching based domain adaptation. arXiv preprint arXiv:2006.13352. — [arXiv:2006.13352](https://arxiv.org/abs/2006.13352) find
  9. Kunal Dahiya, Nilesh Gupta, Deepak Saini, Akshay Soni, Yajun Wang, Kushal Dave, Jian Jiao, Prasenjit Dey, Amit Singh, Deepesh Hada, et al. 2023. Ngame: Negative mining-aware mini-batching for extreme classification. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data… find
  10. Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li. 2022. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212.10450.
  11. Dheeru Dua, Emma Strubell, Sameer Singh, and Pat Verga. 2022. To adapt or to annotate: Challenges and interventions for domain adaptation in open-domain question answering. arXiv preprint arXiv:2212.10381. find
  12. João Gama, Indrundefined Žliobaitundefined, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. A survey on concept drift adaptation. ACM Comput. Surv., 46(4). find
  13. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056.
  14. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324.
  15. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507–517. find
  16. Xingwei He, Zhenghao Lin, Yeyun Gong, A Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023a. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854.
  17. Zexue He, Marco Tulio Ribeiro, and Fereshte Khani. 2023b. Targeted data generation: Finding and fixing model weaknesses. arXiv preprint arXiv:2305.17804. find
  18. Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2023. Large language models are zero-shot rankers for recommender systems. arXiv preprint arXiv:2305.08845.
  19. Bhavitvya Malik, Abhinav Ramesh Kashyap, Min-Yen Kan, and Soujanya Poria. 2023. Udapter–efficient domain adaptation using adapters. arXiv preprint arXiv:2302.03194. find
  20. Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi-Yu. 2023. Active learning principles for in-context learning with large language models. arXiv preprint arXiv:2305.14264. find
  21. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  22. Abhinav Ramesh Kashyap, Devamanyu Hazarika, Min-Yen Kan, and Roger Zimmermann. 2021. Domain divergences: A survey and empirical analysis. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1830–1849… find
  23. Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in nlp—a survey. arXiv preprint arXiv:2006.00632.
  24. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  25. Marco Tulio Ribeiro and Scott Lundberg. 2022. Adaptive testing and debugging of NLP models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3253–3267, Dublin, Ireland. Association for Computational Linguistics. find
  26. Jon Saad-Falcon, Omar Khattab, Keshav Santhanam, Radu Florian, Martin Franz, Salim Roukos, Avirup Sil, Md Arafat Sultan, and Christopher Potts. 2023. Udapdr: Unsupervised domain adaptation via llm prompting and distillation of rerankers. arXiv preprint arXiv:2303.00807. find
  27. Christopher Schröder, Andreas Niekler, and Martin Potthast. 2021. Revisiting uncertainty-based query strategies for active learning with transformers. arXiv preprint arXiv:2107.05687. find
  28. Burr Settles. 2009. Active learning literature survey.
  29. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. arXiv preprint arXiv:1906.08976.
  30. Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2020. Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. arXiv preprint arXiv:2010.08240.
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  32. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
  33. Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. 2022. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering. find
  34. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108.13487.
  35. Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. 2023. Llm-powered data augmentation for enhanced crosslingual performance. arXiv preprint arXiv:2305.14288. — [arXiv:2305.14288](https://arxiv.org/abs/2305.14288) find
  36. Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2022. A survey of active learning for natural language processing. arXiv preprint arXiv:2210.10109. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bansal-2023-large-language-models-annotators-enhancing/