📄 The Linguistic Structure of English Web-Search Queries

Summary

Although web-search queries are widely recognized for their brevity, their grammatical structure has rarely been investigated. The authors examine whether part-of-speech tagging can be applied to English queries and whether it helps search. They sampled Yahoo! queries from August 2006 and lower-cased, spell-corrected and Penn-tokenized them. Then they designed a reduced tag set of 19 classes and hand-annotated several thousand queries. The annotation shows that queries are dominated by nouns, especially proper nouns, while verbs and function words are rare. Most queries form noun phrases rather than loose lists of words. A Brill tagger trained on the labeled queries performed much better than taggers built from conventional corpora such as Brown. The authors also ran initial retrieval experiments. POS-based features ranked high among the features of a machine-learned ranking system, though they did not significantly improve overall relevance. POS tags also predict which words users swap when they reformulate a query. The work matters because it treats queries as their own language variety, with its own distribution and syntax, rather than as degraded sentences.

Key points

• Data: 3,283 sampled queries, of which 2,508 were unique, totalling 8,423 tokens. Tagger experiments used 10-fold cross-validation on 1,602 and later 2,503 labeled queries.
• Tag distribution: The tag set has 19 classes, compared with roughly 90 in Brown.
  • Proper nouns make up 40.2% of tokens, and nouns plus proper nouns about 71%.
  • Verbs account for only about 2.35% and determiners about 0.66%.
  • In Brown, the most frequent tag (noun) covers only about 13% of tokens.
• Inter-annotator agreement: 79.3% at token level and 65.4% at query level, with Cohen's kappa of 0.714 and 0.641. Of 99 conflicting token labels, 69 were judged to reflect genuine ambiguity rather than annotator mistakes.
• Capitalization: Only 16.8% of 290,122 queries contained any capital letters, and usage was inconsistent, so case is a weak cue for proper nouns.
• Retrieval experiments:
  • A query-level "contains a noun or proper noun" feature ranked in the top 13% of ranking features.
  • A feature measuring query–document POS agreement for shared words ranked in the top 12%.
  • Neither feature yielded a statistically significant relevance gain.
  • Substitution probability across reformulations is highest for numbers (0.49), then adjectives (0.46), nouns (0.42) and proper nouns (0.39).

Interesting ideas

• I1. Coarse tag set for fragmentary text: The authors shrank the tag set to the classes that actually occur in queries, because rare fine-grained tags made annotation harder without adding useful distinctions. They kept a simple mapping to conventional tag sets.
• I2. Sorting disagreements into error versus ambiguity: An adjudicator reviewed each label conflict and decided whether it was a mistake or a case where several readings are valid. This showed that most disagreement came from ambiguity, not sloppy annotation.
• I3. Reformulation behaviour reveals core versus modifier words: How often users swap a word between consecutive queries, broken down by POS, separates stable meaning-bearing words (proper nouns) from adjustable slots (numbers, adjectives).
• I4. POS agreement as a matching signal: Instead of only matching identical words between query and document, the authors checked whether each shared word carries the same tag in both. This cheaply filters out matches where the word has a different sense or function.
• I5. Not relying on capitalization: Because case in queries is rare and inconsistent, the authors suggest that proper-noun detection should draw on external lexicons or entity recognition instead.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Titles, headlines and short questions could get a coarse tag layer with a deterministic mapping back to full UPOS/XPOS, so fragment-like inputs are tagged reliably without losing compatibility.
• I2 (learn): When witness parsers disagree on silver data, conflicts could be labeled as error or genuine ambiguity, and ambiguous items kept with multiple accepted analyses instead of forcing one gold answer.
• I3 (answer): When matching a question to text, nouns and names could serve as fixed anchors, while numbers and adjectives are treated as variable slots to check or vary, which makes the matching logic explainable.
• I4 (answer): For QA over fables and school texts, a passage could count as supporting a question only when shared lemmas also share their tag and dependency role, an inspectable check that avoids bag-of-words matching.
• I5 (tokens): For informal or lower-cased input, the tagger could consult WordNet/Wikidata-derived name lists before trusting case, which reduces proper-noun errors that later damage entity tracking.