Elamur Research⌕
☰
home / papers / 2024 / Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

Paper · 2024 · Leonie Weissweiler, Abdullatif Koksal and Hinrich Schutze

Summary

Construction Grammar holds that some meaning belongs to constructions rather than to individual words. The caused-motion construction (CMC) is a classic example: in it, a verb like "sneeze" takes on a motion sense and an extra argument. The authors ask whether current LLMs grasp this, and how to collect enough data on such a rare phenomenon. They propose a cost-aware hybrid pipeline. Dependency parses first filter sentences against the CMC's syntactic pattern. GPT-3.5 then classifies the survivors through prompting, and only the candidates it flags go to expert human annotators. The result is 765 verified non-prototypical CMC sentences, plus an automatic expansion of 127,955 likely instances. To test models, each one is asked whether the direct object physically moves, first with the original verb and then with "throw" substituted. GPT, Gemini, Llama2 and Mistral models all perform poorly. The work offers a reusable recipe for corpus building on rare phenomena and points to the syntax–semantics interface as an LLM weakness.

Key points

  • Pipeline: dependency-based filtering → prompt-based GPT-3.5 classification → manual verification. Each stage raises the share of true instances, which lowers the cost per confirmed example while keeping the data diverse.
  • Released resources: a manually verified CMC corpus of 765 sentences, and a semi-automatically annotated corpus of 127,955 sentences derived from semantic regularities of the CMC.
  • Evaluated models: GPT-3.5, GPT-4, Gemini Pro, Llama2 (7b, 13b, 70b quantised; base and instruction-tuned) and Mistral (7b, 8x7b; base and instruction-tuned).
  • Results: the best model, Mixtral 8x7b instruction-tuned, reaches only 69.75% consistent "yes→yes" answers (30.25% errors), with Mistral 7b IT close behind at 68.12%. GPT-4 scores 57.07%, and GPT-3.5 and Gemini Pro score about 43%.
  • Qualitative finding (GPT-4): sentences answered correctly in both conditions contain mostly prototypical motion verbs (fling, chuck, pull). Answer flips cluster on less motion-like verbs (steal, eat, tie).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓Munich Center for Machine LearningMunich · Germany57 works · 1,094 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Leonie Weissweiler, Abdullatif Koksal and Hinrich Schutze. 2024. Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena. arXiv:2403.06965.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.06965 · arXiv 2403.06965

@misc{weissweiler2024hybrid,
  title = {{Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena}},
  author = {Leonie Weissweiler, Abdullatif Koksal and Hinrich Schutze},
  year = {2024},
  doi = {10.48550/arxiv.2403.06965},
  eprint = {2403.06965},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.06965}
}

References

35 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  2. Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. CoRR, abs/2001.08435. — [arXiv:2001.08435](https://arxiv.org/abs/2001.08435) find
  3. Giulia ML Bencini and Adele E Goldberg. 2000. The contribution of argument structure constructions to sentence meaning. Journal of Memory and Language, 43(4):640–651. find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  5. Noam Chomsky. 1993. Lectures on government and binding: The Pisa lectures. 9. Walter de Gruyter. find
  6. William Croft. 2001. Radical construction grammar: Syntactic theory in typological perspective. Oxford University Press on Demand. find
  7. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  9. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056.
  10. Adele E.. Goldberg. 1995. Constructions: A construction grammar approach to argument structure. University of Chicago Press. find
  11. Adele Eva Goldberg. 1992. Argument structure constructions. University of California, Berkeley. find
  12. Ole Magnus Holter and Basil Ell. 2023. Human-machine collaborative annotation: A case study with gpt-3. In Proceedings of the 4th Conference on Language, Data and Knowledge, pages 193–206. find
  13. Haerim Hwang and Hyunwoo Kim. 2023. Automatic analysis of constructional diversity as a predictor of efl students’ writing proficiency. Applied Linguistics, 44(1):127–147. find
  14. Jena D. Hwang and Martha Palmer. 2015. Identification of caused motion construction. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 51–60, Denver, Colorado. Association for Computational Linguistics. find
  15. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  16. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  17. Bartłomiej Koptyra, Anh Ngo, Łukasz Radliński, and Jan Kocoń. 2023. Clarin-emo: Training emotion recognition models using human annotation and chatgpt. In International Conference on Computational Science, pages 365–379. Springer. find
  18. Kristopher Kyle and Hakyung Sung. 2023. An argument structure construction treebank. In Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023), pages 51–62, Washington, D.C. Association for Computational Linguistics.
  19. Bai Li, Zining Zhu, Guillaume Thomas, Frank Rudzicz, and Yang Xu. 2022. Neural reality of argument structure constructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7410–7423, Dublin, Ireland. Association for Computatio… find
  20. Kyle Mahowald. 2023. A discerning several thousand judgments: Gpt-3 rates the article+ adjective+ numeral+ noun construction. arXiv preprint arXiv:2301.12564. find
  21. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. find
  22. Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. Automated annotation with generative ai requires validation. arXiv preprint arXiv:2306.00176.
  23. Siyao Peng and Amir Zeldes. 2018. All roads lead to UD: Converting Stanford and Penn parses to English Universal Dependencies with multilayer annotations. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 167–177, Santa F…
  24. Josef Ruppenhofer, Michael Ellsworth, Myriam Schwarzer-Petruck, Christopher R Johnson, and Jan Scheffczyk. 2016. Framenet ii: Extended theory and practice. Technical report, International Computer Science Institute. find
  25. Jaromir Savelka and Kevin D Ashley. 2023. The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts. Frontiers in Artificial Intelligence, 6. find
  26. Micah Shlain, Hillel Taub-Tabib, Shoval Sadde, and Yoav Goldberg. 2020. Syntactic search by example. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 17–23, Online. Association for Computational Linguistics. find
  27. Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak, and Petar Milin. 2020. CxGBERT: BERT meets construction grammar. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4020–4032, Barcelona, Spain (Online). International Committee on Computational Linguistics…
  28. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy L… — Gemini: A Family of Highly Capable Multimodal Models
  29. Tiago Timponi Torrent, Thomas Hoffmann, Arthur Lorenzi Almeida, and Mark Turner. 2023. Copilots for Linguists: AI, Constructions, and Frames. Cambridge University Press. find
  30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  31. Yu-Hsiang Tseng, Cing-Fang Shih, Pin-Er Chen, Hsin-Yu Chou, Mao-Chang Ku, and Shu-Kai Hsieh. 2022. CxLM: A construction and context-aware language model. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6361– 6369, Marseille, France. European Language Resources As… find
  32. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. find
  33. Transactions of the Association for Computational Linguistics, 7:625–641. find
  34. Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022. The better your syntax, the better your semantics? probing pretrained language models for the English comparative correlative. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing…
  35. Danni Yu, Luyang Li, Hang Su, and Matteo Fuoli. 2023. Assessing the potential of llm-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apologies. International Journal of Corpus Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weissweiler-2024-hybrid-human-llm-corpus-construction/