📄 GPT系列大语言模型在自然语言处理任务中的鲁棒性

Summary

Large language models perform well on standard benchmarks, but it is less clear how they cope with realistic noise and perturbations in their input. The authors examine this for OpenAI's GPT-3 and GPT-3.5 families: code-davinci-002, text-davinci-001/002/003 and gpt-3.5-turbo. They use 15 public datasets covering 9 NLP tasks, grouped into classification, reading comprehension and information extraction. The test material totals roughly 147,000 original samples.

Each input is altered with 61 linguistically motivated TextFlint transformations at the character, word and sentence levels. Models are prompted zero-shot and few-shot with three prompt templates per task. Fine-tuned BERT serves as the reference.

The GPT models reach BERT-like accuracy on sentiment, semantic matching and reading comprehension. They are weak at NER and relation extraction, where they sometimes invent labels. Robustness is low overall, especially for classification and sentence-level changes. Newer model versions and more demonstrations raise accuracy but do not close the robustness gap.

Checks on an updated gpt-3.5-turbo, gpt-4, LLaMA2-7B and LLaMA2-13B support these conclusions. The study is useful to anyone relying on LLMs for extraction or for noisy real-world text.

Key points

• Benchmark scope: 9 tasks and 15 datasets, including:
  • SemEval2014-Laptop/Restaurant, IMDB, MNLI-m/mm, SNLI, QQP, MRPC and WSC273;
  • SQuAD 1.1/2.0;
  • WSJ (Penn Treebank POS), CoNLL2003, OntoNotesv5 and TACRED.
• Perturbations: 61 TextFlint transformations. The baseline is supervised BERT (BERT-base-uncased; BERT-Large-ITPT for IMDB; BERT-BiLSTM-CRF for WSJ).
• Zero-shot results: gpt-3.5-turbo and text-davinci-003 match or exceed BERT on some datasets, for example SemEval2014-Restaurant and WSC273.
• Error analysis:
  • On CoNLL2003, most NER errors involve confusing entities with the "O" tag.
  • On TACRED, "no relation" cases are often given specific relations, and fabricated relation types appear.
  • On binary IMDB, the models produce an unrequested "neutral" label.
• Model iterations and prompting: across versions, accuracy improves steadily, but robustness gains appear only for sentiment analysis. Changing demonstration content helps somewhat but does not resolve the underlying problem.