📄 Improved Automatic Keyword Extraction Given More Linguistic Knowledge

Summary

This paper addresses automatic keyword extraction from scientific abstracts. The task is framed as supervised classification: every candidate term is labelled either keyword or non-keyword. Earlier systems (Turney 2000; Frank et al. 1999) used stemmed n-grams of at most three tokens, described them with purely statistical features, and required the user to fix how many keywords to return.

Hulth works with 2 000 Inspec abstracts annotated with indexer-assigned keywords. She compares three ways of generating candidates:

• n-grams;
• NP-chunks from a partial parser;
• word sequences matching POS tag patterns that occur frequently among training keywords.

The usual features (term frequency, collection frequency, relative position of first occurrence) are extended with the POS tag(s) of the candidate. Because the classifier makes binary decisions, the system decides on its own how many keywords each abstract receives.

Chunks yield the best precision and patterns the best recall. Adding the POS feature sharply reduces wrong assignments for every candidate type, and the best F-score reaches 33.9. The study shows that inexpensive linguistic information clearly helps a task where statistics alone perform poorly.

Key points

• Corpus: 2 000 English Inspec journal abstracts (1998–2002, Computers and Control, Information Technology), split into 1 000 training, 500 validation and 500 test documents.
• Gold standard: only the uncontrolled keywords were used. 76.2% of them appear in the abstracts, compared with 18.1% of the thesaurus-restricted controlled terms. The test set contains 3 816 present keywords, about 7.63 per document.
• POS patterns: the pattern approach uses 56 POS tag sequences, namely those occurring at least ten times among training keywords; 51 of them include a noun tag.
• Best results on the test set:
  • n-grams with the tag feature: F-score 33.9 (precision 25.2, recall 51.7);
  • chunks with the tag feature: highest precision, 29.7 (F 33.0);
  • patterns without the tag feature: highest recall, 66.0.
• Effect of the tag feature: n-gram output drops from about 42 to about 16 assigned terms per document, while the number of correct terms stays similar.

Interesting ideas

• I1. Learn candidate patterns from the gold data. Instead of hand-writing term grammars, gather the tag sequences of known keywords and keep the frequent ones. This transfers easily to languages or domains where no term grammar exists.
• I2. Let binary classification set the output size. Labelling each candidate yes/no, instead of ranking by probability, removes the arbitrary "top-k" or threshold choice. Each document then gets as many keywords as the model finds justified.
• I3. Use the syntactic shape of a candidate as a feature. A single categorical POS feature acted as a strong filter against bad candidates. It is cheap to add to any span-classification task.
• I4. Measure candidate-generation loss separately. Reporting what share of gold keywords the selection step already misses (about half for chunks, 12.5% for patterns) makes the recall ceiling of each pipeline explicit before any learning happens.
• I5. Treat exact-match evaluation as a strict lower bound. The author points out that indexer keywords are a harsh reference. Multiple annotators or user-based judgments would give fairer scores, which applies to any evaluation of generated or extracted labels.