📄 InstructGPT在命名实体识别任务中的表现和挑战

Summary

Large language models are usually tested on open-ended generation, and their use for structured extraction has received little study. This paper examines InstructGPT on named entity recognition (NER) and compares it with fine-tuned smaller models: BioBERT for biomedical data, RoBERTa for flat general-domain data, and BERT for nested data. The authors test nine datasets covering biomedical and general domains, English and Chinese, and both flat and nested entities.

In zero-shot mode they compare a baseline prompt with a structured prompt that specifies the output format. In few-shot mode they compare in-context learning with chain-of-thought prompting, which they describe as its first use for NER.

InstructGPT stays well below the fine-tuned models. It reaches about 11–56% of their performance zero-shot and at most around 72% with demonstrations. Error analysis shows that many outputs are invalid, either because they break the expected format or because they name spans absent from the input. Invalid outputs also distort evaluation in both directions, so fixing validity alone may not raise scores. The work offers a baseline and a diagnosis for future LLM-based extraction research.

Key points

• Datasets:
  • Biomedical (from BLURB): BC5CDR-disease, BC5CDR-chem and BC2GM.
  • General flat: CoNLL03, OntoNotes 5 (non-numeric types only), MSRA and OntoNotes 4.
  • Nested: ACE04 and ACE05.
  • Each evaluation uses up to 1,000 test sentences.
• Few-shot setup: runs use k = 1, 5 and 10 demonstrations, and each result is averaged over 5 runs.
  • Going from 1 to 10 demonstrations raised F1 by about +10.3 on average for in-context learning and +41.6 for chain-of-thought.
  • With a single example, chain-of-thought scored about 25.2 F1 lower than in-context learning, because of parsing failures.
• Chain-of-thought rationales:
  • Biomedical rationales come from Wikipedia definition sentences.
  • General-domain rationales follow the TriggerNER idea of explanatory trigger words and phrases.
• Invalid generation: roughly 10–70% of outputs (close to half overall) are invalid. When invalidity is severe, spuriously correct predictions can reach about 40%.
• Nested entities: InstructGPT produces far fewer nested entities than the datasets contain, and fewer than half of its nested predictions are correct. Zero-shot nested NER reaches only about 20–29% of fine-tuned performance.